Ajax controversially had a goal disallowed in the first half following a VAR review, but Hakim Ziyech was on the scoresheet in the 75th minute, levelling the scores after Karim Benzema had made the breakthrough on the hour.
Los Blancos were not finished, however, and scored a second through Asensio late on to ensure that they would take an advantage into the second leg of the round-of-16 tie at the Bernabeu on March 5.
Matthijs de Ligt - believed to be a target for a host of the world's biggest clubs – started in central defence, while there was also a spot for Frenkie de Jong, who is heading for Barcelona this summer.
As for Madrid, the main team news centred around Gareth Bale, with the Welshman given the nod in a wide area. Vinicius Junior kept his spot down the left, though, with the teenager outstanding in recent weeks. There was no Raphael Varane, however, as an illness ruled the Frenchman out of contention.
Ajax - backed by a vocal home support - made the faster start to the match and indeed had the first shot inside five minutes, although Ziyech's effort was always clearing Thibaut Courtois's crossbar.
The Dutch giants are present in the round of 16 for the first time since the 2005-06 campaign but had never won at this stage of the Champions League entering the match. There was nothing nervy about the hosts early on, though, with right-back Noussair Mazraoui firing just wide in the ninth minute during a fast start.
Vinicius had Madrid's first serious chance in the 14th minute, but the Brazilian's curling effort was saved by Andre Onana, before the Ajax goalkeeper kept out Bale's header from the resulting corner.
Tadic has been outstanding for Ajax this season and had a golden chance to make the breakthrough in the 26th minute, but the attacker's effort from inside the box clipped the outside of the Madrid post.
Vinicius and Bale both had their moments in the first 30 minutes, but in-form Benzema was struggling to get into the game, with the excellent De Ligt marshalling the Frenchman in a defensive area.
Not since the 2002-03 campaign have Ajax reached the quarter-finals of the Champions League, but Erik ten Hag's side went unbeaten in the group stages of the competition, drawing both home and away with Bayern Munich.
Ziyech should have sent Ajax ahead 10 minutes before the break when he had the freedom of the Madrid box, but the attacker saw his attempt excellently saved by Courtois.
Courtois made a mess of a header from De Ligt shortly after, though, and that led to Ajax having the ball in the back of the net with Nicolas Tagliafico rising to head home. A VAR review ultimately saw the goal chalked off, however, with Tadic, who was offside, adjudged to have fouled Courtois, which led to huge groans inside the stadium.
As it transpired, the first period ended goalless, but that did not begin to tell the story with a hugely entertaining opening 45 minutes suggesting that there would be more excitement to come after the break.
An open start to the second period saw both teams commit players forward on the break, but the final ball often went astray, with Bale guilty of failing to find a teammate from a couple of dangerous positions.
David Neres had Ajax's first real chance of the second half in the 52nd minute, but the number seven could not find a route past Courtois, who stood tall to make the save.
The noise inside the stadium was deafening as the match approached its hour mark, but the home supporters were silenced in the 60th minute when Benzema sent the away side ahead.
The goal came from the left, with Vinicius driving into the Ajax box before feeding Benzema, who picked out the top corner with a stunning effort for his 19th goal of the campaign.
The response from Ajax was strong, with Neres and Ziyech both embarking on dangerous runs, but Tadic was finding it increasingly difficult to make his mark in a central position.
Courtois kept out a free kick from Lasse Schone in the 71st minute, before Dolberg entered the field for the hosts. The substitute's first touch took him into the Madrid box, but he fired high over the crossbar.
Ajax were level in the 75th minute, however, when Ziyech managed to guide a low cross from Neres past Courtois. It was no more than the home side deserved and they immediately went in search of a second.
Dolberg again fired over the crossbar after breaking into a dangerous position, but Madrid were also having chances, with Asensio just missing the post from outside the home side's box.
It appeared that neither team were happy with a draw as both continued to throw players forward late on, with Asensio particularly dangerous for Los Blancos in a central position.
Therefore it did not come as much of a surprise when it was Asensio who put his name on the scoresheet in the 87th minute. Indeed, Carvajal drove down the right before delivering an excellent cross into the Spaniard, who simply could not miss from close range.
Dolberg had yet another chance to score for Ajax late on, but the striker slipped before kicking into the gloves of Courtois, with the defeat meaning that the Dutch giants have it all to do in the second leg next month.
AJAX (4-3-3): Onana; Mazraoui, De Ligt, Blind, Tagliafico; Schone (Dolberg 73'), Van de Beek, De Jong; Neres, Tadic, Ziyech
REAL MADRID (4-3-3): Courtois: Carvajal, Nacho, Ramos, Reguilon; Kroos, Casemiro, Modric; Bale (Lucas 61'), Benzema (Asensio 73'), Vinicius (Mariano 81')