Don't think wind is a problem for lavender. Shade is though. So. Would not put
Lavender there.
So you have a shady free draining site
What,about sarcoccoca....sweet box? Fragrant and evergreen and easily,pruned. Daphne too?
I think I would plant some ferns, a daphne, a sarcoccoca, a couple of hellebores, a geranium or two and a couple of astrantias
If you have more sun than you so far have suggested it opens up prospect of plantIng say a Potentilla for summer colour