You might just need to tigten the BB cups or the crank bolts, I had exactly the same as what you're describing once and that solved it.
@Festerban started
London Fixed Gear and Single-Speed is a community of predominantly fixed gear and single-speed cyclists in and around London, UK.
This site is supported almost exclusively by donations. Please consider donating a small amount regularly.
You might just need to tigten the BB cups or the crank bolts, I had exactly the same as what you're describing once and that solved it.