Oussama Assaidi's early goal was cancelled out by Tom Adeyemi's equaliser before second-half goals from Peter Crouch and Marko Arnautovic looked to have sent Mark Hughes's side through to the fifth round.
© Getty Images
On-loan Liverpool winger Assaidi scored the first goal of the evening's round of fixtures after 10 minutes when he hit a shot from outside in the box into the top corner.
However, the home side hit back when a quick break away resulted in Adeyemi firing past Thomas Sorensen in the Stoke goal on 28 minutes.
Wade Elliot was sent off on the stroke of half time for violent conduct, leaving the home side to play over 75 minutes with a man less.
© Getty Images
The visitors got their noses in front when Assaidi crossed for Crouch who nodded home to score his second goal in as many games 10 minutes after the restart.
Arnautovic then put daylight between the sides with 20 minutes left when he swept home from Steven Nzonzi's cross.
Crouch nearly added a second when his acrobatic effort rattled the woodwork with 10 minutes remaining.
Lovenkrands began his side's fightback on 85 minutes before completing a normal time comeback in the 90th.
Jones's strike in the 94th minute looked to have been enough to take his side through before Lee struck with two minutes left to take the game to penalties.
Nzonzi scored the crucial spot kick to take his side into Wednesday night's fifth-round draw.