I had the same problem with the Swrve trousers - they are just too warm.
I'd thoroughly recommend a pair of Outlier trousers, available in Tokyo Fixed, if your budget can stretch to them. Comfortable, hard wearing and they look fairly smart too.
I was thinking about just buying some M&S type wool trousers off fleabay for winter
I was thinking about just buying some M&S type wool trousers off fleabay for winter