Fresh on the back of dropping their first points of the campaign away to Atalanta a fortnight ago, the Bianconeri squandered a one-goal lead for just a second loss here since January 2013.
Paulo Dybala had a glorious chance to rescue a point for his side with the final kick of the game, only to see his penalty kick kept out in a drama-packed contest.
With Napoli and Roma due to face off at the Stadio Olimpico in the late kickoff, Lazio temporarily occupy second place in the table, sitting above fourth-placed Juve on goal difference.
The initial shot from Sami Khedira was parried into the path of Costa, who was able to blast home from five yards out for his first goal in Juve colours.
Khedira came close once again soon after, this time seeing his shot tipped around the post by Thomas Strakosha as the defending champions went in search of a quick-fire second.
Juve were on top heading into the break and will perhaps feel they should have had a second, as Gonzalo Higuain closed down a clearance and saw the ball ricochet back off the crossbar.
Lazio's fortunes changed after the restart, however, with Ciro Immobile scoring the first of his double after delicately diverting the ball past Gianluigi Buffon - the first goal the hosts have conceded here this term.
Immobile's second arrived seven minutes later, this time converting from the spot following a Buffon challenge on the Lazio striker.
Dybala was introduced from the bench to provide some inspiration, very nearly doing that when sending in a cross that Federico Bernardeschi could not quite turn in.
The Argentina international then struck the inside of the post as the contest edged towards its conclusion, though more drama was to follow deep into added time.
A foul on Bernardeschi initially went unpunished, but the referee then went to the VAR and pointed to the spot to give Juve a chance to rescue a share of the spoils.
Dybala stepped up and saw his attempt kept out by a diving Strakosha, meaning that the Eagles earned victory in this league fixture for the first time since 2003.