The spring in the rear derailleur pulls it towards the smaller sprockets, so what you're describing is a symptom of not enough tension, most likely (as you say) from the shifter slipping.
Assuming it's complete, just do it up tighter.
Which did the trick nicely
@Ste_S started
London Fixed Gear and Single-Speed is a community of predominantly fixed gear and single-speed cyclists in and around London, UK.
This site is supported almost exclusively by donations. Please consider donating a small amount regularly.
The spring in the rear derailleur pulls it towards the smaller sprockets, so what you're describing is a symptom of not enough tension, most likely (as you say) from the shifter slipping.
Assuming it's complete, just do it up tighter.