Upson has revealed that his relationship with the Yeovil boss had become strained during his final months at Huish Park.
"He can say what he wants - if he wants to say that about his top goalscorer, that's fine," Upson told the Southwark News.
"I have got no doubts at all that I have made the step up to the Championship. I've proven myself with my performances and by getting the goals I've got.
"You've got to take things with a pinch of salt and wonder why people say certain things. It's a game of opinions and sometimes you just have to let things go."
Upson joined Yeovil from Ipswich Town in 2010.