Talk to your neighbours, and maybe invite the man in for a cuppa! Try to understand what is wrong with him. He may just be lonely, everyone needs someone to talk too.
You could suggest going out for a coffee or lunch instead of him wondering around by himself, or even the local park?
I can understand that obviously you don't want someone wondering around your yard, so definitely see if there is someone you can contact if it doesn't stop. But i do recommend taking him out somewhere or doing something, because sometimes people just need a friend.
Hope you can sort this out.
Spray him with the garden hose while shouting "No! No!"
Wait, that's for stray dogs.....
Be kind and gentle with the dude.. Talk to your neighbors perhaps they know more..Contact social services when you can't take it anymore.
I justed moved into a new house in the san diego area a few mounths back,
and my neighbor is an elderly man, who is somewhat confused, and sometimes wanders from home, he occaisonaly wanders in to my yard, and will walk around. When he does I will drive him home, and tell him he needs to rest. But it needs to stop, because he doesnt realise what hes doing. I feel bad for him and he doenst have any family what should i do?