The 32-year-old claims to have been targeted by a section of the home fans throughout the match before reporting the situation to the referee and asking that he halt the game.
However, Muntari was booked by the referee for making the request and subsequently walked off the pitch in stoppage time, leaving Pescara to see out the match with just 10 men.
"You saw what happened. The fans were chanting at me during the first half. There was a little kid doing it with his parents standing nearby. So I went over to him and told him not to do it. I gave him my shirt, to teach him that you're not supposed to do things like that. I needed to set an example so he grows up to be nice," Muntari is quoted as saying by La Gazzetta dello Sport.
"Then in the second half it happened in that stand again and so I spoke to the referee. And then he really pissed me off. He told me that I'm not allowed to speak to the fans. I asked him, 'But didn't you hear?'
"I insistently told him he should have had the courage to stop the game. The referee's not just there to stand on the pitch and blow his whistle - he has to manage everything. He should also listen out for that kind of thing and set an example."
Pescara manager Zdenek Zeman added: "Muntari left the pitch because of racist chants, but it's not up to us to dole out justice. We can talk a lot about it but then it must be left with the powers that be.
"We're always talking about racism. Today this has happened when Muntari has already played in Italy for many years. We hope that mentalities will change."
Pescara went on to lose the match 1-0.