"A number of Villa players must look at themselves and ask 'have I, every time, committed myself to the cause?'" Taylor told BBC Sport.
"There was always going to be a group of fans who would never accept Alex. Some of the supporters have moaned and groaned because they feel Alex has set up his teams not to lose as opposed to set them up to win.
"It's an easy thing to say. The money hasn't been available to Alex that had been to Martin. There was a big financial deficit and that couldn't continue. It's been hard for him to make the players get belief in him as to how he wanted them to play."
Villa take on Sunderland tomorrow.