The Samurai Blue were aided by the second-quickest ever red card shown at the finals, as Carlos Sanchez was dismissed inside the opening three minutes for a handball inside the box.
Japan then took control and deservedly sealed their first World Cup win in eight years - and just a fifth ever - three quarters of the way into the contest, with Yuya Osako heading home from a Keisuke Honda corner as the man advantage told.
The drama in Saransk started with referee Damir Skomina reaching into his back pocket after just two minutes and 56 seconds, deeming that Kagawa's shot had struck Sanchez on his upper arm on its way through on goal.
While Carlos Sanchez took the fall, namesake Davinson was equally at fault as he allowed a ball over the top to bounce in the build-up to the penalty incident, giving Osaka a chance to shoot and Kagawa the opportunity to pounce on the follow-up.
Kagawa was the man who stepped up to take the penalty, waiting for David Ospina to dive to his right before sending the ball down the middle.
The first half had an incredibly open feel to it for large parts, with Japan squandering the best of their half-chances on the counter through Takashi Inui.
Colombia struggled for goals in qualifying and, down to 10 men so early in the game and without injured star man James Rodriguez from the off, it was proving to be a frustrating afternoon.
That changed shortly before the interval, however, thanks to a moment of individual brilliance from Quintero.
The River Plate attacker, who had set up two saved shots for Radamel Falcao earlier in the half, sent a free kick under the jumping Japanese wall and inside Eiji Kawashima's post, despite the keeper's best efforts to claw it away.
Japan were unhappy with the referee's decision to award the free kick, though, as they felt Falcao went to ground far too easily once barged into by Makoto Hasebe.
Akira Nishino sent his side out for the second half with a clear message - get at the opposition and make the one-man advantage count.
The Samurai Blue were in complete control for the next 15 minutes and were nearly back in front after Osako got a shot away, which Ospina was equal to on this occasion, as he was when Inui curled one towards the top corner.
With his side struggling to get a grip on the game, Jose Pekerman sent on a half-fit Rodriguez for the final half-hour to no avail.
Colombia came out on top 4-1 when these sides last met in a World Cup group match four years ago, but there only looked like being one winner on the basis of the second half.
That second goal Japan craved arrived in the 73rd minute as, after Hiroki Sakai's close-range shot was blocked behind, Osako met the resulting corner to head his side back in front.
Japan, 45 places below their opponents in the latest FIFA rankings, had to fend off one final attack from Los Cafeteros as James's shot from eight yards was superbly blocked behind by matchwinner Osako.
Victory for the Samurai Blue is their first ever on European soil at the World Cup and their first against South American opposition in the competition.
COLOMBIA (4-2-3-1): Ospina; Arias, D.Sanchez, Murillo, Mojica; C.Sanchez, Lerma; Cuadrado (Barrios 31'), Quintero (James 59'), Izquierdo (Bacca 70'); Falcao
JAPAN (4-2-3-1): Kawashima; Sakai, Yoshida, Shoji, Nagatomo; Hasebe, Shibasaki (Yamaguchi 80'); Haraguchi, Kagawa (Honda 70'), Inui; Osako (Okazaki 85')