The Villa boss has come under fire after seeing his side lose six of their last seven games, including Saturday's 4-1 reversal at the hands of Swansea City.
Holt believes that the players should take responsibility rather than those in the dugout.
"I think it's fair to give the players a bit of stick. The manager has put a team out there to perform and we have not got the points," Holt told Sky Sports News.
"As a group of lads we have got to take the flak and stick for that as we have been beaten 4-1 away from home. We are not in a good vein of form at the moment and we are giving away cheap goals.
"It's always the way when things are going against you that people look at the manager. But we deserve the stick."
Villa currently sit 16th in the Premier League table, three points clear of safety.