Gardening Grandma- I'd certainly establish whose fence belongs to whom. Then legally it's the owners' responsibility to maintain, or not, their property.
We have spent a lot of money over the years maintaining our boundaries. Two back onto non-maintained council 'wild' land, ie now a sycamore grove. A 3rd is an alleyway fence, leading to the land. The fence is our property.
The remaining fence is jointly owned by us & our immediate neighbour. Next to this fence was a mature Crimson King Acer, planted by the original owner of our house, much loved by us & well maintained by a good tree surgeon in the last few yrs. However any overhanging branches from it, & an adjacent Magnolia, used to 'go' every time we went on holiday, without any request, nor returning of the debris. This obviously resulted in lop-sidedness of both trees.
Last yr this neighbour started to grumble in the Spring about the leaves that would fall onto her new patio & lawn, from the Acer. (The magnolia was felled some yrs back, as repeated pruning of them doesnt lead to a good looking tree.) So reluctantly I decided that the Acer should also be felled, which it was in the early Autumn.
It was sad, but the resulting light in the garden, as it was on the South boundary, has been a revelation, even in such a non-summer as this year. I shall miss the extra leaf mould it eventually gave me, & the beautiful outline of the bare branches in the winter, but have no regrets. I know that I stubbornly kept that tree when it was a real case of 'right plant, wrong place'. I had almost 30yrs enjoyment from it & a lot of neighbour hassle!
So I'd advise you to carry on gardening, but be mindful of any possible damage to their property from your plants/trees. Your garden is yours to enjoy & you never know they may even come to appreciate it too. J.