I guess the quick & dirty answer to that question is: the really good ones that are also "small as possible" are much sought-after, thus quite expensive now.
Just search eBay or classifieds for "point and shoot" in a price range you're comfortable with and see what's available.
Slight change of plan! Ended up buying an Olympus 35RC compact rangefinder on eBay for £25. Reasons being that it's got a lot more control than a point and shoot but also has auto mode. I've also never owned a rangefinder and fancied messing about with one!
Can anyone recommend a cheap, small as possible point and shoot 35mm camera to sling in a bum bag?