I would say that in general once they start to flower they are getting close to being ready but it won`t matter if you leave then until the flowers have finished. Once they start to flower see if you can find tubers just under the soil surface. That usually gives an indication of some of the smaller ones. If they are not very big leave them a bit longer. Another tip is to know which variety you grow, i.e. is it a first or second early or a main crop variety as the latter take longer to mature. Another sure sign that they have reached their peak is that the haulm (stem) will start to die back naturally. Hope that helps and good luck.