It's a tricky situaiton but there are a couple of things you can do. Use of the phrase key worker suggests that this person is being supported by adult social care.
I'd advise doing the following. Divide the activities down into three categories; criminal, mental and abusive.
If you think there is criminal activity then call it in on either 999 or 101 as appropriate.
If you think he is exhibiting worrying or dangerous mental behaviour then report it to the appropriate adult social care team..
If you think he is being abused by another person (any kind of abuse; physical, emotional, exploitative, etc) then report it to the adult safeguarding team.
Numbers for adult social care and adult safeguarding (often two different teams) can be obtained from your local authority. If he has a key worker, it may be worth trying to identify and get in touch with them directly as, being immediate neighbours, you could be a good source of information for them.
I realise you want avoid negativity by involving the authorities, however they are in the best position to help you deal with the issues you have with him and support you in your interactions with him. If you don't involve them, there is a greater risk that his behaviour towards you will escalate anyway.
It's a tricky situaiton but there are a couple of things you can do. Use of the phrase key worker suggests that this person is being supported by adult social care.
I'd advise doing the following. Divide the activities down into three categories; criminal, mental and abusive.
If you think there is criminal activity then call it in on either 999 or 101 as appropriate.
If you think he is exhibiting worrying or dangerous mental behaviour then report it to the appropriate adult social care team..
If you think he is being abused by another person (any kind of abuse; physical, emotional, exploitative, etc) then report it to the adult safeguarding team.
Numbers for adult social care and adult safeguarding (often two different teams) can be obtained from your local authority. If he has a key worker, it may be worth trying to identify and get in touch with them directly as, being immediate neighbours, you could be a good source of information for them.
I realise you want avoid negativity by involving the authorities, however they are in the best position to help you deal with the issues you have with him and support you in your interactions with him. If you don't involve them, there is a greater risk that his behaviour towards you will escalate anyway.