I spent a couple of months living in a flat with some fairly extreme mould, which manifested in me developing hives on my skin when exposed to the cold. Given that this was in the alps in winter, it was not ideal. Symptoms stopped as soon as I moved out.
a few years ago we found mould in our old flat and that year was the sickest year i'd ever had - about 6 or 7 colds, even in the summer. Bad stuff.