After some initial pressing by the home side in the opening stages, Timo Werner had a great chance to take the lead in the eighth minute, but his effort from eight yards missed the target.
To compound a frantic first 15 minutes, a Stuttgart penalty appeal was then turned down when Werner's shot struck Rafinha's arm in the box.
The home side eventually took the lead just before the half-hour mark, as a loose ball fell to Vedad Ibisevic, who then side-footed a curling shot into the bottom corner.
Five minutes before the break, Thiago's dipping half-volley was acrobatically tipped over by Ulreich.
Ibisevic very nearly doubled his side's lead just after the restart, following on from a Dante error, but the striker poked his effort wide of the far post.
He would come to rue his missed opportunity, as Claudio Pizarro was able to jump high to head the ball low into the net from a Bayern free kick. Despite Ulreich getting a hand to it, he could not keep the ball out.
Just as the home side thought they had secured a draw, Thiago then broke Stuttgart hearts in the 93rd minute with an overhead scissors volley following a cross from Rafinha.