I have a similar problem but with horses in the field behind me. One of them strayed into my garden last year & did quite a bit of damage - fortunately we were in the middle of the really messy stage of landscaping & we were able to make things right without too much effort.
The landowner is a bit 'strange' and has fallen out with a lot of people in the village and her response has been to just let weeds, hedgerows etc etc grow completely out of control so that neighbouring gardens have a real fight to keep their gardens looking nice. She has been taken to court more than once.
We have been here 3 years & whilst I am not friends with her, we do have a polite & respectful co existence (I think she was relieved I didn't take legal action last year!). We rub along to both our benefits by me approaching her if anything is affecting my garden / view & by her agreeing for me to have the work done. It's not ideal but she is getting work done she should (but wouldn't) do herself and I am getting the job done to my standards & in a way which is most beneficial & pleasing to me.
I would certainly chat to the farmer about this. Does he even know there is a problem? He might agree to do something about it straight away. It might be that he doesn't have the resources to deal with the problem but would be happy for you to sort it out if you have the cash to do it (or he might go halves). If so I would look at strengthening the wall and / or using electric fencing as suggested before.
I certainly wouldn't go down the threatening route except as a final resort.
If the cows don't eat foxgloves they might also resist aconitum (both are toxic) which don't mind a shadier garden.