I usually leave mine till spring. I've a deciduous one in a pot which is currently dying back, so will be moved, when it becomes to unsightly, to a spot out of sight. The evergreen ones are in the ground, dead leaves aren't taken off until the spring for any of them.