Three-time runners-up Celta, who have now won six games in a row in all competitions, took their chances in a lively 10-minute spell in the second half to stun their opponents.
A second successive defeat for Los Blancos, coming after a previous run of nine months without a loss, now leaves them requiring a big performance at Balaidos when the sides face off again next week.
Lucas Vazquez fired away an early shot on goal inside the opening 10 minutes of the game, but it was a slow start to proceedings for the hosts who saw plenty of the ball but created little in terms of clear-cut openings.
There was a penalty shout for Cristiano Ronaldo shortly before half time after being pulled back by Facundo Roncaglia, but the Portuguese was left frustrated at the end of a half that saw him largely feeding off scraps up top.
The second half started in a similar manner, with Madrid struggling to create anything - bar a tame Ronaldo attempt - while Celta were happy to make the most of their counter-attacking opportunities.
Then came a frenetic spell in the match that would determine its outcome, as Iago Aspas latched on to a cross and fired into the back of the net, just moments before Marcelo responded with a volley that deflected off Roncaglia on its way through.
Los Blancos struggled to build on that goal, however, and instead they were soon behind again when Jony was played through and tucked the ball past Kiko Casilla for what proved to be the winning goal of the first leg.
Casemiro came close to bagging a second leveller for his side and Karim Benzema also squandered a big chance, but Madrid had relied on long shots too often and in the end they failed to find a response.