Which components actually make a difference?

Yup, frame play a huge roles, BQ did a blind test by asking a builder to build 3 identical looking frame in three different tubing to see if it's possible to notice the difference.

alias I forget what was the conclusion is, will look through my back issues to find it.

Here's an old one from 1996. Guy rides 7 bikes all the same apart from the grade of tubing; can't tell much difference between them; concludes that fit and geometry are far more important than tube material.

http://www.habcycles.com/M7-8.jpg

Sorry for big image scans of magazine article.