Devonta Freeman got Atlanta back into a strong field position again later in the second quarter, but the visitors were unable to find the endzone again, resulting in another kick for Graham.
Winston and the Tampa Bay offense came back on the field with a minute remaining in the first half, but the quarterback threw an interception, giving the Falcons possession in field goal range, to bring Graham on for another three points, handing Atlanta the lead for the first time.
White and Freeman ensured that the Falcons made a promising start to the third quarter, but they had to rely on the kicker again, with Graham adding a 29-yard field goal to extend Atlanta's lead further.
Tampa Bay edged back in front soon after though as a big catch from Vincent Jackson set up Martin for a one-yard run to the endzone, but a two-point conversion attempt failed.
Connor Barth extended the Bucs lead with a field goal before the end of the third quarter, but Nick Williams found himself open at the start of the fourth quarter, leaving Ryan to launch a touchdown pass to the wide receiver, putting the visitors ahead again.
A five-yard drive from the Buccaneers got the hosts back into a strong field position as they needed a field goal to tie the game, but Winston attempted to get the ball to Evans, who was able to reach above his defender to haul in the touchdown.
The Falcons tried to get back down the other end of the field quickly, but Ryan tried too hard to connect with Julio Jones, throwing an interception to hand Tampa Bay the win.
The defeat means the Falcons can no longer catch the Panthers, who are yet to play this weekend.