It looked like staying that way until full time too, but in a game full of high-quality goals Toivonen saved the best until last as he marked his 50th cap by scoring the winner in incredible fashion with the last kick of the match following a dreadful error from Hugo Lloris.
The result sees Sweden climb above France to the top of Group A on goal difference, three points clear of third-placed Netherlands.
The visitors were able to include the likes of Paul Pogba, Antoine Griezmann and Dimitri Payet in their starting XI, but their superstars were unable to light up a dull and uneventful opening to the contest.
Payet had the first hint of a chance after 14 minutes when he latched on to Benjamin Mendy's cross from the left flank, but it was a difficult shot to take on and the former West Ham United man fired his attempted first-time volley well over the crossbar.
It proved to be the only moment of any note in the opening half-hour, but Sweden finally had their first shots of the contest shortly after the 30-minute mark as first Durmaz and then Mikael Lustig fired wide of the target in quick succession.
That seemed to spark France into life and the first clear chance of the game fell the visitors' way with 10 minutes left until half time when Giroud glanced a header from Mendy's wicked cross wide of the target under pressure from Andreas Granqvist.
Both the Swedish skipper and Giroud required treatment for a clash of heads following that incident, but the Arsenal striker was soon back on his feet and within seconds of returning to the pitch he provided a moment of magic completely at odds with the previous 37 minutes of action.
A cross from the right was only flicked on as far as Giroud, who teed himself up with his first touch before firing the sweetest of volleys into the far top corner for his 17th goal in his last 17 international starts - including six in his last three.
It was a goal worthy of winning any game, but the lead lasted just six minutes before Sweden levelled things up with a well-taken goal of their own as Durmaz fired a first-time strike into the bottom corner after a cross had travelled all the way through to him.
The increased tempo and action spilled over into the second half too, and Griezmann came close to restoring France's lead eight minutes after the restart when his sharp movement inside the box saw him latch on to Moussa Sissoko's cross, only to plant his header straight at Robin Olsen.
France continued to apply the pressure in the opening stages of the second half and Pogba - who scored for Manchester United in the Europa League final at this stadium last month - was the next to come close with a fine long-range effort which flew narrowly wide of the target.
Sweden responded to that period of French pressure with a good spell of their own, though, and Jakob Johansson fired a swerving drive just past the post from 30 yards before Lloris was called upon to prevent Durmaz's cross/shot from sneaking in at the near post.
Olsen was just as busy as his opposite number, though, and the Sweden keeper needed to get across and push away Griezmann's 30-yard free kick shortly after the hour mark to keep the scores level.
Sweden appeared to have an answer to France in every aspect, though, and they came close with a free kick of their own moments later when Lloris tipped Emil Forsberg's dipping strike over the top.
It was France who were applying the late pressure as they went in search of a win that would have taken them six points clear at the top of the group, but Thomas Lemar snatched at two presentable chances after the ball had fallen kindly to him on the edge of the box.
It appeared as though any hope of a winner had evaporated when a late France corner was cleared, but having gone all the way back to Lloris the Tottenham Hotspur keeper then came out of his area and only succeeded in sending his clearance straight to Toivonen.
The former Sunderland man still had a lot to do, but he calmly stroked the ball into the empty net from inside his own half, with Lloris unable to get back in time to atone for his error as Sweden grabbed all three points at the death.
It proved to be the last meaningful kick of the match as Sweden celebrated leapfrogging their opponents in the group, taking control of the one automatic qualification spot for next summer's World Cup in Russia.
The defeat is France's first of their qualifying campaign and leaves them second on goal difference, now only three points clear of Netherlands in third.
Les Bleus will now turn their attention to Tuesday's glamour friendly with England at the Stade de France, while Sweden take on Norway in Oslo on the same night.