-
It also depends what type of Tory you're talking about, as with most things there's a range of opinions.
There are lots of open boarder free marketers who'd advocate entry to all with the motivation to work. Others just think it would be useful to keep track of people coming in/out and deport criminals and those without work. Then you roll through to point-based visa folks, all the way down to those who'll probably support any policy that sticks it to Johnny Foreigner.
My gut says that that people with a conservative mindset dislike change. Immigration represents change. The Conservative party in a large part represents traditionalism.
Although to be fair the only full on outspoken English racists I've ever met irl were all Labour voters.
Then in the 80's they spent a lot of time campaigning for the death penalty for Nelson Mandela....