Buddleias are tough as old boots so don't worry about doing too much. They grow, as I'm sure you know, a lot each year. When pruning, try to imagine how tall you would like the plant to be next year when it flowers. Then look at the main stems and cut them at a height that would get you to your desired height if the plant grew, say, four feet next year. So if you want it to be six feet high next summer with all its flowers doing their thing, and it grows four feet a year, cut it to about two feet up from the ground. Then trim away any silly little straggly side shoots that are just blocking up the centre of the plant and there you are, job done. The more you cut off the more the plant will react with growth next year. If you just cut off a few inches, it will only respond in the same way.
Prune it in the autumn and then the birds will have a chance to eat the seeds first.