No, please don't, or you will have a good hard set of concrete like soil you cannot do anything with!! Do remove the largest from the top, (and keep them for drainage at the bottom of any pots you might devide to use) and then ignore - even if you did remove the ones you can see, there is an everlasting fountain of stones in the middle of the earth theat sends them up all the time - or, more likely, worms, and all sorts of other beasties with whom we share our gardens. You do need some stones for drainage, and provided they are not so large that the mower will not go over them when you lawn has rooted, then ignore all but the largest. If you want to grow carrots or other root veggies you would like to have straight roots rather than winding around a stone, use deep containers, these you can keep stone free.
If yu take a drive through certain parts of the Fenlands you will see that nearly all fields there (and probably other parts too, but these are the ones I know), have a good crop of stones, and produce much of the food we use locally.