Hi Jack, I took over a very neglected garden 40 years ago and am STILL fighting with brambles, seedlings of horse chestnut,hazel nuts,rowen, bind weed couch grass etc. As it is often said on this forum, plants we do not like or ones in the wrong place do annoy us, and, we can easily become obsessed with destroying them. I have dealt with this by taking the advise of an elderly gentleman who lived next door. He used to say "...my dear, don't waste your energy on unimportant things, put it into doing and enjoying what you love doing." Good advice I think, as even today I am still trying to get rid of these plants by lopping and pulling at them; but give most of my time to the ones I enjoy nurturing.
So Jack, I am that elderly person now, and I say to you, do not stress over what annoys you but concentrate on what gives you pleasure. By all means do spend time on clearing what is annoying you, but not so much that summer is gone with you not smelling the roses