Posted: 15/10/2016 at 17:06
The fence is your property; the builder cannot damage your property. He may contest its location; he may also insist, if the deeds confirm his assertion, that you remove it.
When you bought the property, the solicitor who did the conveyancing for you should be able to advise you regarding boundaries.
Stand your ground for the time being, and don't let the builder bully you. He may be worried about future sales of adjoining properties: lack of fencing can make a garden seem much larger than it actually is, and future buyers may be deterred by having a visual statement of the actual size. (I'm guessing here - I have no idea of the builder's true motivation, of course).