Two goals from Eric Lichaj - the second a stunning 20-yard volley - and penalties from Ben Brereton and Kieran Dowell capped an excellent performance from the Championship outfit, who are currently managed by caretaker boss Gary Brazil.
Per Mertesacker and Danny Welbeck both got on the scoresheet for Arsenal, but the Premier League side were outfought in the East Midlands as Arsene Wenger suffered his first ever defeat in the last 64 of this competition.
However, the game was not without controversy with Arsenal claiming that Dowell's spot kick - which came at 3-2 - should have been disallowed due to the Everton loanee touching the ball twice, although replays of the incident proved inconclusive.
The first opportunity of the game went to Forest as Brereton broke in behind the Arsenal defence, but the youngster's low effort with his weaker left foot was easily dealt with by David Ospina.
Up the other end, Welbeck directed a header wide of the post after meeting a cross from Theo Walcott, but it was Forest who continued to have the better of the opening stages and they took the lead in the 20th minute.
Moments after Matty Cash had been denied by Ospina, Lichaj met a free kick from Dowell to power a header into the roof of the net from no more than a yard.
However, three minutes later, Arsenal were level. After Rob Holding had seen a header strike the post, the ball fell to Mertesacker who kept his composure to volley in the rebound from six yards.
Arsenal began to dominate the action but Brereton was a constant threat in Forest's attack and the youngster almost netted a stunning individual effort after wrong-footing Mertesacker only to fire his resulting effort too close to Ospina.
The game looked set to be level at half time but, one minute before the break, Lichaj scored one of the goals of the weekend with the defender controlling the ball on his chest on the edge of the area before firing the ball on the volley into the far corner.
Arsenal needed a response after the restart but, although Wenger's side dominated possession, they were unable to create any opportunities on goal before the game reached the hour mark.
That only gave confidence to Forest and out of nothing, the home side were awarded a penalty when Holding tripped Cash and Brereton took on responsibility and sent Ospina the wrong way.
Arsenal appeared to be out of ideas, but with 11 minutes left Welbeck got his side back into the game after Smith fumbled the ball when challenging the forward, allowing Welbeck to roll a shot into an empty net.
The Gunners were in the ascendancy but Forest were given the chance to seal the win when former player Armand Traore advanced into the penalty area before being brought down by Mathieu Debuchy, although the defender appeared to play the ball.
The drama did not stop there, with Dowell being accused of hitting the ball twice when slipping over as he sent the ball into the roof of the net, but despite Arsenal's appeals, the goal was given.
Joe Worrall was sent off for Forest during the closing stages, but it mattered little as Forest comfortably held on for a huge victory in the context of their season.
NOTTINGHAM FOREST (4-5-1): Smith; Lichaj, Mancienne, Worrall, Traore; Osborn, Vaughan, Cash (McKay 87'), Clough (Walker 56'), Dowell (Mills 93'); Brereton
ARSENAL XI (3-4-2-1): Ospina; Debuchy (Akpom 87'), Mertesacker, Holding; Nelson, Willock (Nketiah 65'), Elneny, Maitland-Niles; Walcott, Iwobi; Welbeck