Milan conceded an early Walter Samuel header, and despite having a man advantage for much of the second half, Allegri's side slipped to a disappointing 1-0 defeat.
Allegri told reporters at the post-match press conference that the referee cost his side at least a valuable point with a series of poor decisions.
He said: "I never talk about referees. But, I have to now. The referee got it wrong in this game.
"In the first half he made a technical error. He blew the whistle for half-time 12 seconds early. Then there was a last-man foul on Urby Emanuelson which was given against him.
"This is not an excuse and perhaps today the referee was unlucky, but the incidents were really obvious."
Allegri also believes that Valeri disallowed a perfectly good goal from Ricardo Montolivo.
He added: "I have no idea what the referee whistled for on the Montolivo goal. Let's just say he was unlucky on the decisive incidents, as the goal would've been crucial."
Defeat to Inter is a bitter blow for Allegri, who is under severe pressure at Milan after losing three of his first four home games this season.