A tight affair at the Memorial Stadium looked like going all the way to penalties until Chris Lines popped up with the winning goal in the 115th minute.
The hosts had the first shot on target of the night when Lines found a yard of space, but Simon Moore was equal to his effort in the Cardiff goal.
The Championship outfit narrowly escaped falling behind shortly before the half-hour mark when Matty Taylor nodded Daniel Leadbitter's cross back into a dangerous area, but the ball was just too far in front of Stuart Sinclair and the chance went begging.
It continued to be Rovers who looked most like scoring, and James Clarke came close with a low strike from range having brought the ball out of defence and been urged to shoot.
The visitors did end the half on top, though, and Joe Ralls came close twice before the interval when first he fired a long-range free kick into the side-netting before seeing another strike deflect narrowly wide of the target.
Chances were tough to come by in the opening exchanges of the second half, but things started heating up in the final 20 minutes when Ollie Clarke fired just wide from outside the area.
Craig Noone responded by coming close himself having picked up a pass from Ralls, cutting inside and curling an effort inches past the post.
Anthony Pilkington was the next to threaten with a low strike from inside the box, but Taylor and Clarke also came close for the visitors in the final 10 minutes.
Neither side could break the deadlock in normal time, though, as the match went to an additional 30, and the first chance of the extra period fell the way of Cardiff when Frederic Gounongbe nodded over the bar following good work from Declan John and Pilkington.
Rovers had Mildenhall to thank for keeping the scores level just before half time in extra time when the keeper produced a fine save to deny Pilkington's powerful strike.
Gounongbe then squandered another chance for the visitors before Rovers threatened a winner themselves when Taylor was denied by Moore.
Cardiff were forced to end the game with 10 men after Stuart O'Keefe picked up an injury when all three changes had been made, and that proved to be costly as Rovers got the winner just four minutes later through a powerful long-range strike from Lines.
The visitors were inches away from rescuing penalties with only two minutes remaining when Gounongbe latched on to Pilkington's pass, only to see his effort come back off the post.
That proved to be the Bluebirds' last chance as they crashed out at the hands of their League One opponents.