Na, don't be nervous about them. I took my bike (totally unsuitable steel gravel muncher) to swinley forest a couple of weeks ago, handlebars took a hammering and wear no problem, but my frame snapped in half! Honest! I've got some ACOR ones, I think from sjscycles.
You can buy some shims for the handlebars, I use them on both bikes and never had a problem. A bit fiddly but excellent accessory to have as you get some right bargain handlebars in the old 26.mm version. About £5-6.
Stupid distance commuter and #supplelife enthusiast.
Why race when you can ride.........but I hate being beaten at the same time.