Doubts

Your experience at the moment combines unfamiliarity with one gear, toe straps, fixed PLUS the complication of panniers - it's no wonder you feel uncomfortable.

Hehe - good point! I'm out of practice on fixed, and am used to SPD pedals, which are easy to get on/off, so toe clips were very unpleasant.

As others have said, get a flip-flop hub and try SS for a while...

I imagine that would be hard work! My experience of fixed has been that the momentum helps one climb the hills in a gear that is otherwise too high on a geared bike. Without the fixed wheel, I imagine going up hills could be very tough on, eg a 70 inch gear.

I have never tried riding fixed with loaded panniers but I imagine it could be a bit unnerving going downhill!

Yeah - it was/is. Maybe I'd get used to it, or maybe that's science's way of telling me to do something different.