I also had this once and it turned out the headtube was out of alignment. Hopefully this isn't the case with your build.
@vpCogworks started
London Fixed Gear and Single-Speed is a community of predominantly fixed gear and single-speed cyclists in and around London, UK.
This site is supported almost exclusively by donations. Please consider donating a small amount regularly.
I also had this once and it turned out the headtube was out of alignment. Hopefully this isn't the case with your build.