Not necessarily - my bosses have/had an incredibly modern house in quite an historic area (hint, it's spitting distance from Norths); sneakily built over the shell of the former property, which was itself built in the 50's over the shell of the property original to the site. Lots of clever architect reasoning behind it, I imagine, that legitimised the whole thing.
http://www.themodernhouse.net/sales-list/the-lawns-south-grove-london-n6/history-857/