There's also the extra muscle damage. I stopped going to the gym during big races because my body has enough work to do repairing legs. If your arms and core are also caned then you've gotta spread the love and that takes more time. So, it becomes a question of goals - bike racing or both?
I don't box but do karate and a hard hours session (particularly sparring) can take it out of you. It's full on anaerobic exercise.