Thing is, your garden is the peak of perfection as it is at present. Nothing more perfect in March than a well prepared bed. It has all the perfection of a blank sheet of paper. You are just like an artist who doesn't want to put the paint on the canvas. But a well prepared bed of soil won't be perfection in June, so off you go! Joy of annuals is that you can't go wrong in a long term sense.
There is another category of gardener, and I am one, who buys plants/seeds/takes cuttings/accepts gifts long before the planting area is cleared and prepared. We would really relish a canvas like yours, instead of the head-scratching that accompanies having the plant, but not the place.