I'd recommend looking back at all the other times people have thought 2 wheelsets 1 bike was going to be the solution. Mostly they end up with one or other set gathering dust in a cupboard because, even as simple as it is, swapping wheels before going for a ride is too much hassle.
I have just that. Is OK but I'd rather have two different dedicated bikes if I had the room, finding time for riding is hard enough without having to change your bike setup
I'd recommend looking back at all the other times people have thought 2 wheelsets 1 bike was going to be the solution. Mostly they end up with one or other set gathering dust in a cupboard because, even as simple as it is, swapping wheels before going for a ride is too much hassle.