Many also believe that Hart wasn't at his best last season, but Milner feels that the doubters are being far too harsh on the stopper.
"He's a victim of his own success," Milner told reporters. "People said he had a poorer season last season but he won the Golden Glove.
"When we played Borussia Dortmund last year [in the Champions League] it was one of the best goalkeeping performances I've ever seen. We could have lost that game 8-1 very easily and that's not exaggerating.
"I am blessed as a midfielder because if I miscontrol the ball I've got four defenders and a goalkeeper to save me. If Joe makes a mistake it's going to end up in the back of the net. That's the unfortunate thing about being a goalkeeper. But he is a top keeper and we're lucky to have him."
Both players are in the England squad to face Moldova and Ukraine in upcoming World Cup qualifying matches.