Gravel / Gravé / Gnarmac / Groad / ATB - AKA Just get a red squirrel

There is a compromise. You're only considering rolling resistance. 42mm tyres are significantly less aero than 23mm and this will be a not insignificant drag at higher speeds.

And if one's using aero rims, there's none optimized for such fat tyres.