Free the Swindon One!

Tottenham fans call themselves by the y word, and that's not generally seen to be racist, because they are re-appropriating a term of abuse that was directed at them because of the large proportion of Jewish fans they had.

But of course if you're calling somebody else it, not yourself, then it's not re-appropriation, it's just abuse.
That is a pretty good point, hadn't really thought of it that way. In this instance though, I think the second point you made is more appropriate.