The visitors took the lead in the first quarter through the boot of Shayne Graham, who kicked two field goals from 31 yards and 50 yards respectively to give his side a 6-0 lead.
They extended that early in the second quarter as Khiry Robinson found his way through a pile of bodies for the Saints' first touchdown of the new season.
© Getty Images
Matt Bryant added a field goal for the hosts as time expired in the first half, and they reduced the deficit to just three in the third quarter as Levine Toilolo was found by a short Ryan pass at the front of the endzone.
Brees saw a pass intercepted later in the quarter, and the Falcons took full advantage to move into the lead for the first time.
Ryan threw a short pass to Antone Smith, who did the rest to complete a 54-yard score with just 21 seconds remaining in the third quarter.
The Saints refused to let their heads drop, however, and were soon back in front courtesy of a three-yard Mark Ingram run in the fourth quarter.
There were yet more twists to come, though, as Atlanta regained the lead with just under three minutes remaining of the match thanks to a 17-yard Jacquizz Rodgers run.
The visitors responded yet again as Ingram rushed in for another three-yard score to put his side ahead with just over a minute left, but Bryant converted a 51-yard field goal as time expired to force overtime.
Bryant was the hero again in the additional period, this time slotting his kick over from 52 yards to hand Atlanta a dramatic victory.