They always peck at the holes, it's a habit they have but not sure of the reason. They will eventually enlarge the hole, but oddly that means larger birds can get in, which makes the box unsuitable! (some things about nature are not easy to work out!). It doesn't mean the holes are not the correct size to start with though.
One can only guess your boxes are not quite right for them for some other reason. I face the hole to the north and put them on fairly long sticks well away from cats in trees or bushes. Avoid busy areas such as those used by lots of birds to get to feeders, they won't choose a box where they have to compete all the time with other birds It's often wise to make your own from the dimensions given out by wildlife trusts etc, they usually work as they have been tried and tested many times. The plans for the ones I've done are here: