Madrid led 2-1 from the first leg in Amsterdam, but Ajax put in a stunning performance in the Spanish capital to reach the quarter-finals of the European Cup for the first time since the 2002-03 campaign.
The defeat means that Los Blancos have now lost each of their last four home games in all competitions, including twice to Barcelona in their two fixtures before what proved to be a historic evening at the Bernabeu.
The 13-time champions have been semi-finalists or better in each of the last eight seasons, but they will not have the chance to play for a fourth straight European Cup, which is expected to lead to the exit of head coach Santiago Solari.
Matthijs de Ligt - believed to be a target for a host of the world's biggest clubs – started in central defence, while there was also a spot for Frenkie de Jong, who is heading for Barcelona this summer.
Madrid should have taken the lead in the fourth minute when Lucas delivered a wonderful cross onto the head of Raphael Varane, but the Frenchman's effort crashed off the crossbar from close range.
It was Ajax that made the breakthrough in the seventh minute, however, with Hakim Ziyech finding the bottom corner with a super finish after meeting a cross from Tadic, who had been presented with the ball by a loose Toni Kroos pass.
Ajax goalkeeper Andre Onana had to keep out an effort from Vinicius Junior before Kroos struck wide in the period that followed, but the visitors were dangerous on the break with David Neres and Ziyech causing issues either side of Tadic.
Incredibly the Dutch giants doubled their advantage in the 18th minute when Tadic danced past a number of challenges before sliding an excellent pass into Neres, who rounded Thibaut Courtois before lifting the ball home.
Neres should have made it 3-0 in the 22nd minute when he found himself inside the Madrid box once again, but the attacker's effort just drifted wide of the far post, before Vinicius was denied by Onana down the other end.
Bale was introduced in the 29th minute of the match, but it was not a tactical change, rather an enforced one as Lucas limped off the field with an injury.
Vinicius remained a threat for Los Blancos and fired just wide in the 32nd minute, but that proved to be the Brazilian's final piece of action as he also picked up an injury while striking the ball and was replaced by Marco Asensio shortly after.
Ajax had another opportunity 10 minutes before the break following a mistake from Bale, but Courtois came to the home side's rescue as he kept out a strike from the lively Ziyech.
Bale came close to registering in the 42nd minute when he broke into the Ajax box before attempting to convert, but the Welshman's effort hit the post, which allowed Ajax to enter the half-time break 3-2 ahead on aggregate.
As expected, the whistles inside the Bernabeu were deafening, with the home fans showing their displeasure at what had been a quite incredible opening 45 minutes of football in the Spanish capital.
Asensio curled just wide of Onana's post early in the second half as Madrid desperately looked for the goal that would have levelled the contest at 3-3.
Ajax continued to carry a threat, though, with Courtois being forced to keep out a low strike from Donny van de Beek in the 51st minute during an incredibly open period.
Benzema missed the Ajax post with a curling strike in the 59th minute, before Noussair Mazraoui hit the side-netting down the other end of the field.
Ajax had a third in the 62nd minute, though, when Tadic found the top corner with a stunning effort. There was quite a long delay in the aftermath while VAR checked to see whether the ball had crossed the line earlier in the move, but the goal was given and Madrid were left needing three in the final 25 minutes or so to advance.
Onana made a smart save to deny Bale 20 minutes from time, but Madrid did register shortly after when Asensio's low strike drifted into the bottom corner of the net.
Ajax answered straight back, though, with Lasse Schone's free kick dipping over Courtois in the 72nd minute to all but end the contest, which led to a number of Madrid fans leaving the stadium.
Bale was denied by Onana late on as the holders looked for a second, but it finished 4-1 on the night, with Ajax, who passed up a big chance for a fifth late on through Ziyech, advancing to the final eight courtesy of a 5-3 aggregate success.
There was time for it to get worse for Madrid, however, with Bale picking up ankle injury late on while Nacho was sent off following an altercation with Van de Beek.
REAL MADRID (4-3-3): Courtois; Carvajal, Varane, Nacho, Reguilon; Casemiro (Valverde 88'), Kroos, Modric; Lucas (Bale 29'), Benzema, Vinicius (Asensio 35')
AJAX (4-3-3): Onana; Mazraoui (Veltman 80'), De Ligt, Blind, Tagliafico; Van de Beek, De Jong, Schone (De Wit 74'); Ziyech, Tadic, Neres (Dolberg 74')