The Central American side, now led by coach Oscar Ramirez, reached the quarter-finals in Brazil four years ago, but were undone by some memorable quality from Kolarov, whose set piece capped off a functional showing from the Orlovi in the city of Samaras.
Los Ticos should have gone ahead after 12 minutes when Serbia fell asleep at a corner. The short set piece allowed David Guzman to cross into the middle, where centre-back Giancarlo Gonzalez was left unmarked to somehow head over the bar.
The chances soon kicked Serbia into life and they began to take possessional control of the contest. Branislav Ivanovic showed promising skill on the right to beat his marker, before Keylor Navas did well to get a hand to the resulting cross driven across his goal.
It was down the right channel where Serbia caused issues in a pragmatic first half, this time as Dusan Tadic dragged men wide to release Adem Ljajic, although no Serbian player gambled on his cross into the middle.
Gonzalez's early header proved the best chance of the first period and Serbia's closest effort to match the miss came courtesy of Lazio playmaker Sergej Milinkovic-Savic, who produced a memorable bicycle kick that was palmed away by Navas.
Costa Rica showed more forward endeavour after the break, but it was Serbia who should have taken a lead when Milinkovic-Savic played Aleksandar Mitrovic beyond the defence, although he was thwarted by a brilliant save from Navas.
Serbia's quality in the final third eventually told, but it came in avoidable circumstances when Guzman carelessly brought Mitrovic down on the edge of the area. The set piece allowed Kolarov to step up in the 56th minute and bend a memorable free kick over the wall and beyond Navas into the top corner.
Arsenal forward Joel Campbell was introduced with Costa Rica chasing the game and he almost capitalised on an error from Nikola Milenkovic, but the centre-back recovered remarkably to deny the striker as he found himself one-on-one with Stojkovic.
Serbia should have made the final 10 minutes a little less nerve-wracking when Tadic crossed for substitute Filip Kostic on the penalty spot, although Navas got a vital touch on the ball to prevent the forward from finishing on the half-volley.
Milinkovic-Savic turned in a standout second-half showing with some memorable forward passes and he played in Mitrovic to put the game to bed, but the forward fluffed his lines under pressure from Gonzalez.
In the end the misses were not too matter too much, though, as Mladen Krstajic's charges held off some late Costa Rican pressure in order to seal three important points.
Potentially tougher clashes against both Brazil and Switzerland now await Serbia later in the group.
COSTA RICA (5-4-1): Navas; Acosta, Gonzalez, Duarte, Calvo, Gamboa; Borges, Venegas (Bolanos 60'), Guzman (Colindres 73'), Ruiz; Urena (Campbell 66')
SERBIA (4-2-3-1): Stojkovic; Ivanovic, Milenkovic, Tosic, Kolarov; Matic, Milivojevic; Tadic (Rukavina 83'), Ljajic (Kostic 70'), Milinkovic-Savic; Mitrovic (Prijovic 90')