Albania managed to keep the tournament hosts out of the door when Didier Deschamps's side found their shooting boots in the second half, but Griezmann, who started on the bench, netted in the 90th minute and Payet rounded off the win in five minutes of additional time.
Considering that one of the favourites were taking on the lowest-ranked team in the tournament, many would have tipped France to race out the blocks in the first half.
That was not the case, however, as only a handful of opportunities were created by the side selected by manager Deschamps, who opted to drop Paul Pogba and Griezmann in favour of Kingsley Coman and Anthony Martial.
Olivier Giroud, who scored in Friday's 2-1 win over Romania, could have added to his tally in the fourth minute when Payet's free kick found him in the box, but the Arsenal striker's header went over the crossbar.
Albania did not muster many attacks either, but they grew in confidence in the first half and were given space and time on the ball in possession.
Some decent play in the middle of the park ended with Ermir Lenjani unleashing a 25-yard shot from a central position, but his effort soared into the stands.
Martial's best opportunity of the first half was kickstarted by Payet, who threaded the ball through for the Manchester United attacker, but the youngster opted against a first-time shot, allowing Arlind Ajeti to get a block in.
The forward charged towards the ball, but goalkeeper Hugo Lloris was able to get to there in the nick of time before Sadiku could reach the cross.
There were some shaky moments in defence for France, who were almost undone by a training-ground move when Ledian Memushaj faked an in-swinging corner by drilling it low to Lenjani, whose close-range strike was blocked.
After the break, France changed their shape and tactics as Pogba replaced Martial, which resulted in Payet moving to the left, and the team pressed higher with more energy.
Coman almost scored instantly after the restart when Giroud delivered the ball into the box from the left-hand side. It was helped on by Blaise Matuidi towards the Bayern Munich player, whose six-yard header bounced wide.
Albania still caused a threat at the other end, and they came close to finding the opener when a ball into the box deflected off Bacary Sagna and onto the post.
Minutes later, Pogba came close when he got on the end of Payet's ball towards the back post, but the Juventus star's close-range half-volley fizzed over the bar.
France piled on the pressure as the second half continued, with Giroud producing two headers in quick succession, but one bounced wide of the goal and the other was denied by the post.
The hosts' pressure eventually paid off when Griezmann netted in the 90th minute with a header into the corner of the net from 10 yards out.
Five minutes later, Payet scored his second goal in as many games with a bending shot past the keeper to send France into euphoria.
Les Bleus are at the top of Group A, while Switzerland sit two points behind in second, with Romania and Albania third and fourth respectively.