City took an early lead through Sergio Aguero, but two penalties from Nabil Bentaleb gave Schalke a half-time advantage and City were left further frustrated by the dismissal of Nicolas Otamendi after the break.
However, Leroy Sane came off the bench to score a stunning 30-yard free kick against his former club before Raheem Sterling completed the turnaround in the final minute of normal time to give City an advantage ahead of the return fixture next month.
City dominated the opening stages of the game and their first opportunity arrived in the seventh minute, with Aguero seeing a well-directed header tipped over the crossbar by Ralf Fahrmann.
Kevin De Bruyne also had a long-range effort saved by the goalkeeper, but City took the lead in the 18th minute when David Silva squared the ball for Aguero to tap home from eight yards out after Schalke had lost the ball in their own defensive third.
Schalke felt aggrieved that play had not been stopped for an injury to one of their players, but City remained on the front foot as they targeted more away goals before the break.
However, Schalke improved as the first half progressed and shortly after Mark Uth had curled wide of the far post from 25 yards, the German outfit found an equaliser but in controversial circumstances.
A penalty was awarded after Otamendi was adjudged to have handled a shot inside the area, but the call was only reached after several minutes of indecision by the referee and VAR.
Bentaleb sent Ederson the wrong way from the spot to get Schalke back on level terms, and a second spot kick arrived before the break with Fernandinho hauling down Salif Sane as the defender attempted to get on the end of a set piece.
Former Tottenham Hotspur midfielder Bentaleb found the roof of the net with his second penalty to give Schalke the lead, although Bernardo Silva came close to a leveller for City in the fourth minute of added-on time as his effort from 20 yards just cleared the bar.
City went on the attack after the restart with De Bruyne curling a low effort just wide of the post from 16 yards out, while Aguero and Ilkay Gundogan both lashed shots off target from distance.
Pep Guardiola's side remained in the ascendancy as the game progressed past the hour mark, but Otamendi gifted Schalke a man advantage by receiving a second yellow card for a foul on Guido Burgstaller.
The dismissal did not lead to a change in approach from City, but they were not forcing Fahrmann into much work between the sticks with De Bruyne seeing a deflected free kick go wide and Bernardo Silva sending a strike over the bar from distance.
City appeared content to return to England with a narrow defeat but Sane had other ideas after replacing Aguero, with the German sending a swerving free kick into the right corner via the aid of the post.
Schalke began to look vulnerable after the equaliser and City completed a stunning turnaround in the final minute of normal time to put themselves in full control of the tie.
A long ball from Ederson found the run of Sterling - who nudged Bastian Oczipka off the ball - and the winger kept his composure to find the bottom corner of the net from eight yards out.
SCHALKE 04 (3-5-2): Fahrmann; Bruma, Sane, Nastasic; Caligiuri, Serdar, Bentaleb, McKennie (Skrzyski 78'), Oczipka; Uth (Harit 87'), Mendyl (Burgstaller 65')
MANCHESTER CITY (4-3-3): Ederson; Walker, Fernandinho, Otamendi, Laporte; De Bruyne (Zinchenko 87'), Gundogan, D.Silva (Kompany 70'); B.Silva, Aguero (Sane 78'), Sterling