I dig my massive plot over twice a year and hoe weeds off every week - spring to autumn. Your problem is that once you've dug it , your leaving it and letting the weeds build up again.
Me personally would strip the top layer of weeds off with a spade. Cover the whole lot with weed membrane, not purchased from the pound shop, can I just say. But the real 'I am' horticultural membrane. Then build your self some nice raised boxes, they don't have to be 20ft tall, the height of a plank is sufficent and fill them with a nice mixture of soil/compost. Of course, if you want them 20ft tall thats your prerogative.
A lot less hassle then having to keep turning and improving heavy clay soil - I give up with me mams. The gravel went back down and flower pots were rearranged - nightmare !!