If you consider the stresses the frames will go through with a strong rider on them (I'm not saying you're not) then I think then can handle a few potholes or bumps up curbs. You are more likely to damage the rims than the frame, in my experience. I have an aluminium road bike and havent had any issues.
As provenrad said, thicker tyres will help if you are worried about poor road surfaces etc.
If you consider the stresses the frames will go through with a strong rider on them (I'm not saying you're not) then I think then can handle a few potholes or bumps up curbs. You are more likely to damage the rims than the frame, in my experience. I have an aluminium road bike and havent had any issues.
As provenrad said, thicker tyres will help if you are worried about poor road surfaces etc.