It all depends on your size and type of citrus fruit as some are bred to be more suitable for colder weather nowadays. Also, the older the plant, the more likely it will survive colder temperatures. If you are in a micro climate, sometimes, you can keep the plant against a south or south-west wall, and they can cope, but if it is your first year with this plant, then put into a green house or even indoors somewhere cool away from heating.
It's totally normal to see flowers and then fruits on the same bush/tree. The fruits can take up to a year to mature and some longer. Ideally, to get more fruits, you need the heat and the night temperatures to be a minimum of around 15-16 degrees celsius for half a year. This is quite hard to achieve in the UK, so the reason they are hit and miss.