Fans rounded on McLeish following Tuesday night's 2-1 defeat at home to Bolton Wanderers.
The result has left Villa just three points above the drop zone with three games to play.
"Obviously you can accept the fans' frustrations because we aren't winning games and they're spending good money to come and watch us," Bannan told Sky Sports News.
"We're all in this together, though, so it's unfair to just blame the manager as the players are just as much at fault. It's harsh how the manager is getting treated because as a team we should all take the criticism."
Villa continue their campaign on Saturday with the short trip to West Bromwich Albion.