Yes, all parts of them are toxic, but consider other common garden plants that are partly or completely poisonous - yew, laurels, foxgloves, sweet peas, Delphiniums, Lupins - the list goes on. To be completely on the safe side you should check to see how many plants in your garden are poisonous or have poisonous parts - and remove them. It all boils down to whether your grandson is likely to eat the plant. This is highly unlikely as Oleanders apparently taste AWFUL (do not try this at home!).