Paulo Di Canio's League One outfit dominated the first-half action and established an advantage courtesy of a Collins double.
The summer signing from fellow promoted club Shrewsbury Town opened the scoring on 27 minutes when he capitalised on a Thomas Sorensen error.
The Danish keeper, selected in place of the rested Asmir Begovic, dropped a Jay McEveley cross, allowing Collins to tap into an empty net from close range.
He added another when he headed in a corner from Raffa de Vita, who nearly opened up a three-goal advantage for Swindon when he struck the post in first-half stoppage time.
Tony Pulis made six changes to the starting XI that drew with Arsenal at the weekend but the fresh faces could not reduce the deficit.
The Potters boss introduced big guns John Walters and Peter Crouch in a bid to salvage his side's cup campaign.
It was another forward, Kenwyne Jones, who gave Stoke a way back into the game when he headed in Michael Kightly's cross before substitute Walters forced extra time when he calmly slotted in low from inside the area.
Despite Stoke having the momentum it was surprisingly Swindon who struck first in the additional half-hour as Aden Flint headed in past Sorensen.
However, City restored parity once more when substitutes Walters and Crouch combined, with the former crossing for the latter to nod home.
With the game seemingly heading for penalties, Collins completed a memorable night for both himself and Swindon when he converted the rebound from Ashley Williams's parried shot.
With the result, Stoke joined Fulham in being the only two Premier League teams to exit at the first possible stage after the Cottagers lost 1-0 to Sheffield Wednesday.