Are you using organic flour and is the jar/whatever open to the air, not sealed?
Organic flours will have more mocrobes on (in?) them anyway so a better start for your colony, and not sealing the container will allow microbes from the air to float in and start a party.
Should making a starter be absolutely impossible to fuck up?