The Ferdinand brothers in particular were said to be disappointed by the length of suspensions that Chelsea captain John Terry and Liverpool frontman Luis Suarez received as punishment after being found guilty of racially abusing an opponent. In Terry's case, he was disciplined by the Football Association for abusing Anton.
As a result, they ignored Kick It Out's anti-racism t-shirt launch last October - something that deeply disappointed Carlisle.
"It pisses me off that, rather than speaking out and addressing the topic, a few decide to do something as trivial as not wear a t-shirt, then melt back into the crowd," he wrote in his autobiography that is being serialised in the Daily Mail.
"Rio and Anton Ferdinand were among a group of high-profile abstainers, along with Joleon Lescott, who has not worn one since Newcastle's Turkish midfielder Emre was cleared of racially abusing his Everton team-mate Joseph Yobo, and Jason Roberts.
"Jason apart, the Premier League 'big' players happily throw in a grenade, but do nothing when it explodes, nothing other than appear to hide behind their advisers. That's because they are shithouses. They have made their 'statement' for all to see, but how many are prepared to stand in front of the cameras and be counted?"
Neither of the Ferdinands nor Lescott has yet replied to Carlisle's words.