For what it is worth, I disagree with those who say that there is nothing you can do.
It seems to me that his concentration on that area had to do with the greenhouse; may have thought he was doing his friends a favour. Even if it is only a bluff, I think you should stick to your guns and get him to put up some fencing, if possible at his own expense. It isn't so much about the the money as about establishing that you will not accept being treated in that way. If he won't do it, you haven't lost anything. You don't have to be nasty - just firm.
I wonder if you already have a gut feeling about what to do and just need to follow it.
Anyway, as Bunny says, life is too short to let yourself be too upset. I really hope you get a satisfactory outcome and lots of enjoyment from your garden for a long time to come.