Vermiculite acts as an assistance to good drainage in compost, and can be used to cover small seeds in trays, the kind that need some light to help with germination. I always mix some in the compost I am using for pots, as not only does it help with keeping the soil open and draining well, it holds some moisture within in, so if something gets really dry there is a little moisture for the plant to get at. I have found it better than water retaining gel in baskets, as it does not go into the sticky clumps the gel tends to.
It is in no way dangerous, but as you would not breath in loose compost by waving it around in the air, neither should you do so with anything else - as ever, a little common sense applied works wonders.
Perlite is basically epanded polystyrene, and does much the same job as vermiculite, but I prefer the latter as it is not an oil based product as perlite is. Which one you use is a personal choice.