Costa Rica took the lead seven minutes into the second half as Bryan Ruiz found the bottom corner with his effort, but their task got a lot harder when Oscar Duarte was shown a red card in the 66th minute.
Greece finally found the equaliser through Sokratis Papastathopoulos in stoppage time, but they couldn't make their numerical advantage pay in the 30 minutes of extra time.
Penalties followed and, after the first four players for either side had converted, Theofanis Gekas saw his effort saved by Keylor Navas, allowing Michael Umana to convert the winning spot kick for the Central Americans.
Here, Sports Mole takes an in-depth look at whether Costa Rica deserved to reach the quarter-finals for the first time in their history.
On target: 1
On target: 8
Was the result fair?
It is tough to gauge fairness following a penalty shootout as defeat is cruel on whoever is on the losing side. Costa Rica would have been devastated to concede the equaliser so late into normal time, but in truth Greece deserved that goal as they created the better chances over the 90 minutes.
Extra time was one-way traffic for large periods as Greece peppered the Costa Rican box with crosses, but the Central Americans hung on admirably and did really well to keep the Europeans at bay despite having a man fewer. It would have been harsh on either side to have lost in the added 30 minutes as both put everything they had into it.
The penalties, of course, could go either way and, on this occasion, they went to Costa Rica. The stats are overwhelmingly in favour of Greece, but they are lopsided as a result of the red card, with Costa Rica having to play for almost an hour with 10 men in the end. Greece will feel that they should have won as they had the better chances, but it is hard to begrudge Costa Rica the victory.
Costa Rica's performance
They certainly weren't as fluid and free-flowing as the Costa Rica we saw against Uruguay and Italy. They came into the match in the unfamiliar position of being favourites and, with Greece challenging them to play through them, Jorge Luis Pinto's side seemed to be short of answers for large spells.
The fact that their goal was the only shot on target they had is indicative of how little they threatened throughout, with Orestis Karnezis having a fairly quiet match other than picking the ball out of his net. However, their main strength today was not in their attack, but in their defence. They hung on brilliantly against a Greek siege and the 10 men left on the field after the red card were exhausted by the end.
A quarter-final with the Netherlands now awaits them, and with that comes a return to the role of underdogs. It will allow Costa Rica to revert to their counter-attacking style, which suits them better than trying to break teams down. However, they will need a performance more in keeping with their first two at this World Cup than their last two if they are to progress further still.
For a long time it was the same old Greece as they sat back and showed minimal attacking intent for large spells. They had the best chance of an otherwise uneventful first half as they looked to break quickly or capitalise on long balls forward having stifled the Costa Rican attack.
The opening goal was the best thing that could have happened to the game as it forced Greece to become a bit more adventurous, while the red card merely enticed them to attack even more. They were kept at arm's length by a stubborn Costa Rican defence for a long time, but they finally got the deserved breakthrough in the 91st minute.
Had Keylor Navas not be on such good form then Greece would have won this match fairly comfortably, but they couldn't finish the chances they created and paid for that eventually. They improved as the game wore on and are perhaps unlucky to be going home, but the World Cup is ruthless and Greece learned that tonight.
Sports Mole's man of the match
Keylor Navas: Yet again it was the man between the sticks that made the difference for Costa Rica tonight. Navas made a string of sensational saves to deny Greece, including two from Kostas Mitroglou in the 93rd minute and the 121st minute respectively - both of which would have been winners.
He also made a superb stop to deny Dimitris Salpingidis in the first half, while it was another impressive save that gave Umana the chance to score the winning penalty in the shootout. Navas could have perhaps done better for the goal as he parried the ball back into a dangerous area, but Costa Rica would have comfortably lost this in normal time with a lesser keeper.
It would be harsh to give this to Gekas for his missed penalty in the shootout as it was more a good save than a bad miss, so instead it goes to Oscar Duarte for his sending off. While on a yellow card he dived into a rash challenge that was always likely to earn him a second yellow, and that put his side under immense pressure for the rest of the match.
Ben Williams had an eventful game today, handing out no fewer than eight cards, including the red to Duarte. The sending-off was the correct decision, although a more lenient ref may have given him a second chance, but there was one big call that the Australian got wrong in the second half.
A cross from the right hit the hand of Vassilis Torosidis in the box, an infringement that none of the officials managed to spot. On first glance it did look to have hit his head, but the replays showed that it was quite a clear handball.
Costa Rica: It is uncharted territory for Costa Rica as they take on Netherlands in the quarter-final of the World Cup next Saturday.
Greece: Greece, meanwhile, will turn their attention to qualifying for Euro 2016, with their campaign getting underway against Romania in September.