Whyte had dominated the opening four rounds and twice sent the 40-year-old Russian to the canvas.
But Povetkin turned the fight on its head with a ferocious left uppercut that ended the fight in an instant.
"I didn't feel I would finish the fight like this but I was pretty confident in the fourth round," Povetkin told Sky Sports.
"I went down twice but it was OK. It was not too much damage.
"I was watching his fights and I see that he was missing uppercuts from the left and from the right.
"All my training I trained for those shots and it was definitely one of the best punches of my career."
Whyte, the WBC's number one contender for over two years, knew he could not afford any mistakes behind closed doors at Matchroom HQ in Essex.
The prize of meeting the winner of the trilogy fight between current champion Tyson Fury and Deontay Wilder was on the line.
There was a sedate start to the fight as Whyte began to establish his impressive jab and Povetkin struggled to make an impression.
Povetkin was left off balance by a strong right in the third and Whyte troubled the Russian further as he worked well to the body.
Whyte was getting more confident by the round and Povetkin took a count after being dropped to the floor by a right-left combination early in the fourth.
Povetkin was looking out of his depth and he was sent tumbling to the canvas again by a short left uppercut close to the bell.
It looked as if the end was near for the Russian but he landed a perfect punch from nowhere that left Whyte on the floor for some moments, although he was able to eventually get to his feet.
Fury was quick to comment on Povetkin's victory, writing on his official Twitter account: "40 & proud", while IBF world featherweight champion Josh Warrington added: "Wow! crazy this boxing ya know..."
Whyte's promoter Eddie Hearn, who confirmed there was a rematch clause which he will exercise before the end of the year, said: "I can't quite believe it to be honest. When the punch landed I felt I was in some dream.
"The fight was over virtually after two heavy knockdowns, but this is the drama of the sport we love.
"This is the drama of heavyweight boxing and one punch can change everything."