Olives send down deep roots to seek water in their usually dry environment. That teeny pot will be strangling them and also it will be starving and probably thirsty. The best way for it to thrive is to plant it out in the ground but be sure to shelter it from cold winds and hard frosts as it will like neither.
Prune any obvious dead branches then give the whole pot a good soak to help loosen the rootball and then tease the roots so they can unwind and spread in a new planting home dug a little deeper but two or three times as wide as the pot. Pierce the base deeply with a garden fork to allow for drainage and mix the original soil with some good MPC or garden compost to allow you to plant the tree at its original depth. Fill the rest of the hole the same way, firm the roots gently with your foot and water generously.
Mulch with some more MPC or chipped bark and keep watered this summer until the autumn rains so it can get established without a check.
If you can't plant it out or need to be able to wheel it under shelter for winter, at least give it a pot 2 or 3 times as wide and deep and decent John Innes no 3 compost. You will need to loosen the roots as described and it will be entirely dependent on you for all watering - rain is not enough in a pot in spring and summer - and for an annual spring feed.