Midway through the first half, Rangers' Steven Whittaker received a red card for throwing the ball at a Malmo player in retaliation to a mistimed challenge.
Despite the numerical disadvantage, the visitors went ahead with a Nikica Jelavic strike to lead at the break.
The second-half featured two red cards, one for Rangers' Majid Bougherra, who elbowed an opposition player while challenging for an aerial ball, and the other for Malmo's Ricardinho.
The hosts levelled things on the night with Mamad's strike ten minutes from time to put the Swedish side in the hat for the main competition.
Rangers will now face a qualifier to enter the Europa League group stage.