Magdeburg, who are 10th in the fourth tier of German football, beat Bundesliga opposition in Augsburg in the first round, but got off to a devastating start this evening.
Bayer, fifth in the Bundesliga, were in front inside three minutes when Hakan Calhanoglu picked out the top corner with a 30-yard free kick.
In the rest of normal time, Leverkusen had the lion's share of the chances, but Magdeburg weren't completely on the back foot and had a period midway through the second half where they looked particularly threatening.
Bayer's chances of avoiding an upset were dealt a blow when, 12 minutes from time, Son Heung-Min was shown a straight red card for getting into an altercation.
It looked like it would be the home side heading through when Niklas Brandt slotted the ball in from close range with just over 10 minutes to play in extra time.
However, Magdeburg's lead lasted just a couple of minutes as Kyriakos Papadopoulos nodded in from a free kick at the near post, ensuring that penalties would be needed to split the sides.
The visitors fell behind straight away in the shootout as Emir Spahic saw his spot kick saved, while another missed penalty from Stefan Kiessling left Magdeburg needing to score just one of their last two.
However, the nerves got the better of the hosts, as Brandt and Nicolas Hebisch both had their efforts saved to take the shootout to sudden death.
With Leverkusen leading 5-4, Bernd Leno managed to hold on to Magdeburg's attempt to level, sending the Bundesliga side through.