The defeat for Arsenal means that the Gunners will be absent from the Champions League for a second successive season, and Arsene Wenger's reign as head coach of the London club will end without a trophy.
Atletico boss Diego Simeone made three changes to the team that started the first leg at the Emirates Stadium last week, with Vitolo, Gabi and Costa coming in for Sime Vrsaljko, Kevin Gameiro and Angel Correa.
As for Arsenal, Wenger selected the same XI that started last week. Alexandre Lacazette again led the line with support from Danny Welbeck and Mesut Ozil, with Jack Wilshere, Granit Xhaka and Aaron Ramsey in midfield.
The Gunners had the first half-chance of the match in the fifth minute when Lacazette broke down the right before delivering a dangerous cross into the Atletico box, but there were no takers.
Costa should have sent the home side ahead less than one minute later when he bullied Nacho Monreal off the ball before driving into the Arsenal penalty area, but the striker fired wide of the post.
There was then a significant stoppage in Madrid with Arsenal skipper Laurent Koscielny picking up a serious Achilles injury, which led to the Frenchman being stretchered off the field in the 11th minute.
The next half-chance of the match came for Costa in the 25th minute, but the Spain international waited far too long inside the Arsenal box, which allowed Calum Chambers back to make the clearance.
Lacazette found himself free inside the Atletico box just before the 30-minute mark, but the Frenchman could not sort out his feet in time, and eventually a very promising moment for the Gunners disappeared.
Koke was next to come close when his inventive effort in the 37th minute almost caught out Arsenal goalkeeper David Ospina, but the ball just missed the post, before Antoine Griezmann turned wide of the goal moments later.
It did appear that the two teams would be heading down the tunnel at 0-0, but Atletico had other ideas, with the Spanish giants taking the lead in the second minute of added time at the end of the first period when Costa held off the challenge of Hector Bellerin inside the Arsenal box before finishing over Ospina.
Costa headed into the arms of Ospina early in the second period, before Koke fired wide of the post minutes later as Atletico looked for a second on home soil.
Ramsey had a huge chance for the Gunners in the 53rd minute when the ball broke for the Welshman inside the Atletico box, but he could only bundle wide of the post under pressure.
Costa then had a strike blocked down the other end during an open period of the match at the Wanda Metropolitano.
Ozil started to influence proceedings for Arsenal just before the hour, but the Gunners continued to pass up opportunities as the second period gathered pace. Indeed, Xhaka saw his low strike saved by Jan Oblak, before Shkodran Mustafi headed over the Atletico crossbar.
Costa and then Griezmann had chances for Atletico 20 minutes from time, but neither could find the back of the net as Arsenal survived more scares inside their own box.
Henrikh Mkhitaryan struck just over the Atletico crossbar in the 72nd minute as opportunities continued to arrive, and Costa was causing Arsenal's back four all sorts of problems with his strength and movement.
Fernando Torres eventually replaced the excellent Costa in the 83rd minute, and Atletico managed the game in the final moments to secure a 2-1 aggregate success as Arsenal suffered more heartache.
ATLETICO MADRID (4-4-2): Oblak; Thomas (Savic 93'), Gimenez, Godin, Lucas; Vitolo (Correa 74'), Saul, Gabi, Koke; Costa (Torres 83'), Griezmann
ARSENAL (4-3-3): Ospina; Bellerin, Mustafi, Koscielny (Chambers 11'), Monreal; Ramsey, Xhaka, Wilshere (Mkhitaryan 68'); Welbeck, Lacazette, Ozil