I can understand that for a TT or similar then you're obviously going to select something else but, so far, I haven't found a situation where Rapha clothing hasn't performed as it should (with the exception of their shoe covers which I don't rate.)
I usually cycle around 6,000 miles a year in all conditions so I do care about the performance of my gear although, again, you saying that those who care about performance don't wear Rapha doesn't make sense. RCJLT and Team Sky clearly care as do Rapha Focus.
Rapha does not make the sky tt skin suits