I have garden on all 4 sides but the main area ia to the north which means the sun gets round to the main terrace at about 3pm. Too cool for sitting out except on warm days but usually a breeze of some sort when it's really hot and we get to enjoy the sun till it sets at 10:30 in midsummer as we have no shade from neighbours.
When we've finished sorting the house out we'll ahve a sitting area to the east and to the south can have all year sun for sitting out at veraious times of day.
Each aspect has its virtues - hot sun lovers to the south, soft fruits to theh west for warm, ripening sun, shade lovers up next to the house to teh north and pretty well anything I fancy further away from the house as long as it's bone hardy.. Veggies are currently on the north side in raised beds but have short sun hours in spring and autumn and none in winter so may migrate to the east where they'll get more sun as it's open to the south too..