I can't ride my Speedplay unless I'm 99% sure I wont have to dab. So, dry gravel is fine. Muddy trails anything steep and/or technical (for me) not so much.
I am thinking about trying my SPD SIDIs again though (non-midfoot) but honestly I think going back to flats is the right move.
Flats vs clipless please!