We have been raking off gravel (naturally occurring stuff) from our Veg patches for 16 years and still it comes to the surface.
Except for tap rooted Veg it is not a problem and in many ways it is useful. Believe it or not whilst it is recommended to add gravel to help drainage, it also acts as a moisture reservoire, if the gravel is irregular in shape. Every piece of gravel has a thin film of water around it and the more surfaces it has the bigger the amount of water covering it. Plant roots soon find it which is why you often se gravel stuck to roots.