The visitors, who had already sealed qualification for next summer's tournament in Russia prior to today's match, took the lead after less than four minutes through Thomas Meunier, but Bosnia responded and went into half time ahead courtesy of two goals in the space of nine minutes.
Belgium did the same themselves in the second half to swing the topsy-turvy match back in their favour, including a goal for Jan Vertonghen on his record-equalling 96th international appearance.
Bosnia gave themselves another glimmer of hope when Dario Dumic equalised again with just eight minutes of normal time remaining, but parity lasted only one minute this time as Yannick Carrasco struck the winner immediately afterwards.
The result leaves Bosnia sweating on the result of Greece's match against Cyprus, with one point currently separating Mehmed Bazdarevic's side from Greece in the race for a playoff spot in Group H.
The hosts went into the match knowing that only victory would ensure matters stay in their own hands in that battle for second place, but they fell behind after less than four minutes as Belgium made a perfect start to the match.
Medunjanin came close again midway through the half with a long-range strike which flew narrowly over, before Belgium were forced into an early change as Marouane Fellaini limped off with an apparent knee problem.
The hosts levelled things up seconds after Fellaini's exit when Edin Dzeko squeezed the ball across the face of goal via deflections from Thomas Vermaelen and Courtois, leaving Medunjanin with the simplest of finishes into an empty net at the back post.
Things got even better for Bosnia with just six minutes of the first half remaining when Jan Vertonghen's mistake allowed Edin Visca to race clean through on goal, and the midfielder made no mistake with his finish as he drilled a low strike past Courtois.
It was the first time that Belgium had conceded two goals in a single game of their qualifying campaign, but they responded well and could have restored parity within three minutes but for a miraculous double save from Begovic.
Michy Batshuayi initially turned a low strike towards goal from close range which Begovic did well to keep out, and the Bournemouth goalkeeper then denied Vermaelen from similar range on the follow-up.
Begovic almost turned from hero to villain moments later, though, when he was rescued by the post after almost turning Hazard's low cross into his own net when trying to intercept it.
The visitors picked up where they left off at the start of the second half as Batshuayi flashed one header wide of the near post before Toby Alderweireld was only denied the equaliser by a goal-saving block on the line from Ognjen Vranjes.
Bosnia briefly hit back with a couple of half-chances from set pieces, but Belgium were back level shortly before the hour mark when Carrasco's turn earned him space inside the box. The winger's resulting effort was saved by Begovic, but he could not hold on to the ball and Batshuayi was alert to punish the spill from close range.
The Bosnia keeper was called into action again shortly afterwards when he tipped Youri Tielemans's low drive wide, but there was nothing he could do when Belgium regained the lead with just over 20 minutes remaining.
Vertonghen - who was making his 96th Belgium appearance to draw level with Jan Ceulemans as the most capped player in the team's history - was left open at the back post to atone for his earlier error after a corner had been flicked on.
The visitors should have put the game to bed four minutes later when Batshuayi found himself through on goal, but Begovic denied the Chelsea striker before Carrasco sent a sliced effort looping onto the roof of the net.
Bosnia's need for a win saw them press more men forward in the closing stages and both Vedad Ibisevic and Dzeko squandered decent chances before the equaliser finally arrived through Dumic's close-range header.
The home side's renewed hope was only short-lived, though, as Belgium added the final twist to a thrilling contest with the winning goal just one minute later as Carrasco latched on to Meunier's clever pass before tucking his finish past Begovic at the near post.
Bosnia now realistically need Greece to slip up against Cyprus if they are to keep their hold on second place, with the latter facing Gibraltar at home in their final game while Bosnia take on Estonia.
Belgium, meanwhile, are now 11 points clear at the top of Group H and have won their last 10 World Cup qualifying matches on the road.