I have three Buddleja 'buzz' and in the write-up it says they are ideal for container planting, I don't agree with this and they are far better planted in the ground, so you are definitely doing the right thing. Mine have grown to 5ft and are still in flower, very important to deadhead too.
I have another that is just really a cutting, about 10inches tall with four big flowers and still making more budds. It's in a clay pot but I'm not sure whether to leave it there during Winter or plant it in the ground now. Just to say they are a fantastic species of Buddleja and because they are head height, all the better for seeing the butterflies that love them.