Football

You are right of course. I heard a guy on Five Live the other week saying he witnessed this exact sort of behaviour from some fans at your up-market neighbours.
He made a good point that it should start with the fans self-regulating one another. If you see someone doing this next to you, it is your duty to report it instantly to the stewards. Though some people may be intimidated and too scared to report it.
Once reported however, it is then the responsibility of the stewards to remove them from the stands and have them immediately arrested. It is then the responsibility of the club to ban them for life.

It has to start with the initial reporting though

Chelsea act very quickly when racist abuse is reported to them. I know because I have reported such abuse. I expect that I will be called a grass by many on here but I believe very strongly that it is the duty of everyone to stamp this behaviour out.