They do need lots of good drainage so it sounds as if you have that right. They need as much light and warmth as they can get, the harder you grow them the better the oils in the leaves and thus the better perfume and taste you get.
When we visit our friend in the very south Spain we walk among hot dry dunes and so forth, where the rosemary just is stunningly good - no soil to speak of, no rain for most of the year - the perfume is glorious. Guess that is what they like then.