Had similar problem - turn out that the handlebar was a little bend on one side, not only it's worth checking your handlebar/stem alignment, but also have a careful look at your handlebar if it bend slightly.
@edscoble started
London Fixed Gear and Single-Speed is a community of predominantly fixed gear and single-speed cyclists in and around London, UK.
This site is supported almost exclusively by donations. Please consider donating a small amount regularly.
Had similar problem - turn out that the handlebar was a little bend on one side, not only it's worth checking your handlebar/stem alignment, but also have a careful look at your handlebar if it bend slightly.