The 23-year-old beat Paulie Malignaggi via split decision back in June and Stafford feels that the three-weight world champion does not get enough praise from others in the sport.
"They're always going to be harsh. He's 23, and he's fighting a 33-year-old veteran in [Malignaggi] and they don't give him credit," Stafford told HustleBoss.com.
"That's crazy. The kid is nothing but 23 and doing a spectacular thing.
"They expect too much out of him. And they feel that he doesn't hold up to what they think he's supposed to do and they're saying he's not the one."
Broner himself recently forcefully dismissed talk of a rematch between him and Malignaggi.