My Grandad's garden was heavy clay (so much so I took some into school and my pottery teacher fired it as an experiment), and that had tiny bits of coal in it. Have you tried digging down to find out how deep it goes? Obviously if it's only a couple of inches, then I'd move it, but if the clay goes down more than a foot, it will be one helluva job to shift it.
Personally, I'd see how far down it goes before deciding on anything. If you get fruit trees, you can keep them in pots for a few years, so if there's lots of contamination, shift it in stages, but still have your fruit trees, and hopefully by the time they need planting in deep soil, you'll have fixed your problem.
My Grandad planted some fruit & veg in his garden (after adding LOTS of sand and manure), and was in his 70's when he died, so if there are tiny bits of coal in a heavy clay soil, then I can't see it being a problem, as it didn't affect him (mind you the smoking and drinking he did didn't seem to affect him, and I wouldn't advise doing either)! I don't claim to be a scientist, but I think most of the nasty stuff in coal is released when the coal is burnt (heavy metals and sulphur), so if it's not burnt I don't think you'd have a problem.