Following the game many of Serbia's players were criticised for their performance, although Kolarov has now slammed the "annoying" fanbase.
"I have been here for a while and I can say that we have never had the right kind of support," the Manchester City defender is quoted as saying by The Guardian.
"It seems people in Serbia can't wait for us to falter, so they can insult us. It's annoying, although I don't have a personal problem with it because I know who and what I am. Most of them are probably using their keyboards to vent their own frustrations."
Serbia return to action on Tuesday night when they face Albania.