That's not true. Maybe the crowd you hang out with, but I commute from / in South London daily and a majority does not wear Rapha stuff. Most of my cycling wardrobe is not Rapha either, though that's about to change as I bought some cheap used stuff on Ebay...
You'd stand out like a sore thumb around here (south London) by not wearing Rapha.