Celta were almost ahead in spectacular fashion after just three minutes, but Andreu Fontas's strike from a partially cleared corner rattled the Premier League side's crossbar.
The Saints came back into the game through Jay Rodriguez, who first put in a low cross that was not converted before firing over the bar from inside the box.
The striker than had the best chance of the game to open the scoring on 22 minutes but after getting himself clear, he missed the target with his effort.
Celta came back into the game towards the end of the half but despite this, the sides went in level.
Southampton piled on the pressure late on, with Guly do Prado hitting the bar with a header, but the game ended goalless.
Celta Vigo then won 4-1 in the shootout.