Torres was also deemed to have gone over too easily in search of a penalty whereas Carl Jenkinson won a foul despite minimal contact.
"It is the job of referees to stop them. It's not my job. The referees have to punish them," said Di Matteo. "They did it, but it took some time for them to do it.
"I am against diving, absolutely. It's something we should eradicate from the game. It's probably not easy but if there's anything that can be done we are for it. It's probably complicated."
The Blues are still unbeaten in the Premier League and sit top of the table.