What should have I done?

You shouldn't have given out your address.I'd move house if I were you.
This guy sounds like a nutter.He'll put the frighteners on you.
Hope you're gonna be ok.

+1 I think you did the correct thing more or less but don't give out your address, your phone number in enough.

+1 to Hippy too, if you are going to ride away make it count, don't stop in the first place..