Former Magpies striker Daryl Murphy opened the scoring after less than two minutes, and it looked as though that would be the decisive moment until the bulk of the drama unfolded in and beyond the six allotted minutes of stoppage time.
Newcastle looked to have forced a penalty shootout when Salomon Rondon scored his first goal for the club with a powerful low finish into the bottom corner, but Forest hit straight back and restored their lead just two minutes later, with Matty Cash adding a 94th-minute strike to his 93rd-minute equaliser from the previous round.
Still the match had more twists and turns, though, with the visitors being denied a late penalty on Ayoze Perez before Forest finally put the game to bed just seconds later when Gil Dias lobbed the keeper in the 97th minute.
Forest had suffered from slow starts in a number of their outings already this season, but they were immediately on the front foot this evening and drew the first save from Karl Darlow after only 50 seconds when Dias went for goal from just outside the area.
It proved to be a very early warning shot for the sleepy visitors, and the home side were ahead moments later when Murphy notched his 100th senior club goal against one of his former clubs.
Sam Byram was the architect of the opener, though, skipping past Kenedy all too easily down the right flank before whipping a devilish ball into the box which Murphy glanced past Darlow from close range after less than two minutes.
It made for a perfect start for Aitor Karanka's side, and they maintained a threat throughout the opening 45 minutes with some slick football befitting a team that had spent more in the summer transfer window than their Premier League opponents.
Newcastle did begin to grow into the match, though, and their first hint of a chance arrived after only eight minutes when Jamie Sterry's cross from deep almost picked out Joselu, who could not get enough on his header to turn it goalwards.
Ki Sung-yueng also failed to make the most of a presentable opportunity to work the keeper when he dragged a 25-yard effort wide at the end of a slick team move, but Newcastle's best chance of the first half came with just 10 minutes remaining.
Joselu got in front of his marker from a corner to glance his header right across the face of goal, but his effort dropped narrowly past the far post while both Yoshinori Muto and Federico Fernandez were inches short of tapping it in from close range.
Forest continued to hold their own, but they suffered a blow in the closing stages of their first half when Byram - arguably their most impressive performer up to that point - was forced off through injury.
It took until the opening minutes of the second half for Newcastle to test Luke Steele in the Forest goal for the first time, and even then it came from an offside position as Joselu flicked his header towards goal from Christian Atsu's cross.
The hosts responded immediately, and it took a last-ditch goal-saving block from Fernandez to deny Murphy his second of the evening after Darlow had spilled Dias's effort at the feet of the striker.
Ki scuffed a half-chance for the Magpies shortly before the hour mark, but the best second-half chances continued to fall the way of Forest and Darlow was at full stretch to claw away an audacious chip from Joao Carvalho which threatened to drop under the crossbar.
Dias was the next to come close for the home side when he skipped away from two Newcastle players before bursting forward, but he may have been better served trying to slip in Murphy rather than sending his own effort high and wide of the target.
Karanka may have been beginning to fear repercussions of his side's profligacy, and he watched another chance to put the game to bed go begging 12 minutes from time when Murphy found Dias, who fired narrowly wide when he should have at least hit the target.
The Forest nerves were only heightened by the news that six minutes of stoppage time would be added, and the boos that greeted the fourth official's revelation initially proved prophetic when Newcastle levelled things up in the 92nd minute.
Rondon - on as a sub for Joselu just 10 minutes previously - started and finished a move which also saw Atsu and Perez involved before he drilled his finish into the bottom corner for his first goal since being handed the iconic number nine shirt.
Suddenly it looked as though penalties were inevitable - a result which would have been harsh on the hosts - but having rescued his side in the previous round, Cash was once again the saviour to restore Forest's lead just two minutes after the equaliser.
Dias was released down the left flank and delivered a low cross into the box which Darlow could only parry out into a dangerous area, where Cash was waiting to fire home.
The manic celebrations once again seemed premature moments later, though, as the topsy-turvy finale continued to captivate fans until the final whistle, with Steele clattering into Perez but escaping without any punishment from referee Jeremy Simpson.
It was a decision which left Newcastle seething long beyond the end of the game, and their mood was not helped by Forest adding a third seconds later, Dias lifting a skilful finish over a stranded Darlow to kill the game off once and for all following a madcap period of stoppage time.
The result means that Forest have now knocked Newcastle out of this competition in the second round two years running, leaving the Magpies absent from the third round in successive seasons for the first time since 1983-84.
Rafael Benitez's side also remain winless this season ahead of matches against Manchester City and Arsenal either side of the international break, whereas Forest are still unbeaten across all competitions in 2018-19.
NOTTINGHAM FOREST (4-2-3-1): Steele; Byram (Darikwa 39'), Hefele, Fox, Robinson; Bridcutt, Watson; Lolley (Osborn 58'), Carvalho (Cash 82'), Dias; Murphy
NEWCASTLE (4-4-1-1): Darlow; Sterry, Schar, Fernandez (Perez 69'), Clark; Atsu, Ki, Longstaff, Kenedy; Muto (Murphy 77'); Joselu (Rondon 82')