Yes, it certainly is possible and they like pretty poor, well-drained soil. The only problem with perennial wildflowers is they tend to get going in the second year. You could group the pots together and surround those pots with pots of annual wildflowers until your perennials get going. I mix mine together, annuals and perennials in a border as the annuals self-seed and I pull up what I don't want