Aggression would be worse. Ignorance can be worked on. For the life of me, I still can't see why there aren't a series of educational tv ads to ask drivers to modify their behaviour in relation to cyclists and even ads to highlight cyclists responsibilities too. Modes of behaviour have been affected in this way before with reasonable success.
TfL are planning something along these lines, a campaign to increase the empathy between cyclists and drivers. I hope it works better than the last attempt about 5 years ago that led to lots of "good advice" to cyclists about behaving better.