Antoine Griezmann scored the only goal of the game shortly after the hour mark as Diego Simeone's side made it four points from two games, while newly-promoted Rayo remain rooted to the bottom of La Liga following successive defeats.
However, the visitors will have been much more encouraged with their performance against their Madrid rivals than they were when they began the campaign with a 4-1 home defeat to Sevilla on the opening weekend.
Atletico countered with a corner of their own which culminated in Saul Niguez firing an effort from the edge of the area in the sixth minute which Los Vallecanos charged down.
Michel's charges were by no means overawed by the task at hand as they continued to take the game to their opponents in the early stages of the encounter.
The hosts grew into the game as the minutes ticked on but concrete chances remained at a premium until the 37th minute, when Thomas Lemar and Angel Correa forced goalkeeper Alberto Garcia into a double save.
Neither side could find an opener come the half-time whistle, but Simeone's charges continued their good work after the restart and were rewarded with the breakthrough on 63 minutes.
A corner into the Rayo box was flicked on at the far post by Stefan Savic and into the path of Griezmann, who managed to hold off his marker and rifle home from close range.
Atletico struggled to build on their advantage, though, and they were fortunate to come away with all three points in the end as Rayo squandered two golden opportunities late on.
Alex Moreno was guilty of both of those, with the first coming three minutes from time when he collected the ball inside the box and got his shot away, only to be denied by a fine stop from Oblak.
Oblak was the hero again in stoppage time too, thwarting Moreno from point-blank range when the Rayo man looked certain to rescue a point for his side at the death.
Atletico held on to kick-start their season in their opening home game, though, lifting themselves up to second in the very early La Liga standings.
ATLETICO (4-4-2): Oblak; Hernandez, Godín, Savic, Juanfran (Partey 36'); Lemar, Niguez, Rodrigo, Correa (Koke 66'); Griezmann (Martins 76'), Costa
RAYO VALLECANO (4-2-3-1): Garcia; Moreno, Ba, Amat, Advíncula; Elustondo, Comesana (Pozo 69'); Kakuta (Medran 69'), Trejo, Garcia (S Moreno 77'); Embarba