Not sure what you mean by pot worms, could it be vine weevils eating the roots? Earthworms would not damage living plant parts. I don't know about millipedes which are black often coiled up wormlike things. Sounds like you need expert help, that excludes me but would love to hear if you find out.