Despite making clear contact with his bat, Broad stood his ground after Clarke had taken a smart catch at slip.
Umpire Aleem Dar turned down the confident Australian appeal and, with no reviews remaining, Broad remained at the crease.
"It's up to him, there's no doubt about it," Clarke told Sky Sports News. "I have always been a believer in the umpires are paid to make decisions, that's why they have a job.
"If every batter was going to walk when they hit the ball, we wouldn't need umpires but obviously individual players choose to walk when they hit it. I certainly don't think any less of Stuart that's for sure."