How about a low but open fence?
I also get ants and eggs in my compost. I ignore them, and continue to turn the compost when needed, as if they weren't there.
I used a water and bicarbonate of soda solution to great effect on an aphid infected plant.
My wife has saved a bee today that looked almost dead. I quick supply of sugar water and the bee perked up and walked off. My wife could see it actually drinking it up...
I have a compost thermometer and having mixed fresh grass clipping into my compost, with a thick layer on top for insulation, the temperature quickly gets above 40/50 degrees.
I understand your predicement but it would be a shame to destroy their nest. They will leave at some point but I can't say when. Maybe when they have left you can relocate the box elsewhere so if they return to it they won't be bothering you?
I think Monty might have used rockwool when they rebuilt the pond (lake) at Berryfields a few years ago. I recall they had professionals in to make a clay liner?
Grass cutting do help my compost get really hot. You can store compost anywhere, maybe cover it to stop the rain washing it away?
I have no knowledge of rockwool. Alan Titchmarsh did once laygrass turf so that it overlapped the edge of the pond and went a couple of inches into the water.
Looks like the blackbird nest behind my garage has been abandoned/attacked? She was happy sitting on the nest leading upto my two week holiday. Upon my return home the other day the nest is empty, although there are no signs of broken egg. What a shame.