"Certainly the image of the club has suffered from this issue - there's no denying that," Di Matteo is quoted as saying in the Daily Mail.
"We try to do everything in our power to maintain high standards with the club's employees and supporters.
"People make mistakes. He has apologised and he is being punished. It has put a little bit of a cloud over the image of the club."
However, Di Matteo insists that Terry remains an important part of his team.
"John realises that he let himself and the club down on that day, but over the years he has proved himself to the club," said the 42-year-old.
"I am a football manager - I judge the players, not their lives. We followed the legal process and supported our player.
"I still believe his account of what happened on that day. He has accepted his ban and realised that the language is not acceptable. He is not getting away with it - he has been sanctioned."