The hosts were unable to build momentum after that score though, with Geno Smith finding his range to punish a string of defensive errors from Atalanta.
First the rookie quarterback combined with Jeff Cumberland for the second time in as many games, floating a 20-yard touchdown pass to the tight end out wide.
Falcons quarterback Ryan was then sacked on the next forward drive for the Falcons, allowing Smith to create another first and goal with a huge 47-yard reception by Cumeberland.
Jeremy Kerley got free in the endzone to extend New York's advantage, with the Falcons stopped one-yard short of the line with their final play of the first half to trail 17-7 at the break.
Jacquizz Rodgers quickly ran in a touchdown for the Falcons when play resumed in third quarter, but the Jets responded as Smith used play-action to find Kellen Winslow in the endzone for a 27-14 lead.
With time running out the Falcons came roaring back as Rodgers burst through for his second touchdown of the night, before late pressure told as Ryan found Levine Toilolo in the corner to take the lead with less than two minutes left.
The Jets were furious with the manner in which Toilolo scored after the defence were penalised for holding Tony Gonzalez, but with time running out quickly got within field-goal range.
With three seconds left Smith's drive got Folk the chance to win the game from 43 yards and the kicker duly delivered, sending his effort up the middle to snatch a last-gasp victory.