Lots of garden plants are poisonous to some degree - just be sensible, don't eat anything you don't know is edible, and wash your hands after working in the garden. Remember that these plants have been growing in gardens for hundreds of years and the human race hasn't been wiped out yet. Lots of the plants have medicinal uses - foxglove's latin name is digitalis and it is still used to make medication for some heart conditions.
Rhubarb leaves are poisonous whilst the stalks are edible. The seeds of potato plants are also poisonous but we don't remove potato plants from our gardens! Daffodils are poisonous to cats, but amazingly cats and daffodils continue to coexist in many gardens.
In my experience pets that are used to having the free run of the garden are unlikely to eat toxic plants. Cats that do not have access to gardens may develop the habit of eating houseplants and in doing this they may eat something poisonous.