The R's have made a slow start to the season since returning to the top flight, picking up just four points from their first seven games to find themselves rooted to the foot of the table.
However, Ferdinand says that it is now down to the underperforming players to turn the club's fortunes around in the coming weeks, starting against Liverpool this weekend.
"We are not even in double figures for the number of games played yet," the former England international told the Evening Standard. "To push the panic button now would be crazy in my eyes.
"Harry still comes across to me as a fighter and a man with a lot of fight left in him. He wants to get the club out of this situation we're in. It's not for the lack of trying.
"No one wants to see their manager get sacked. It was not nice to see it happen to David Moyes at Manchester United last season and I wouldn't want to see it here. It's down to us as players, as well as the coaching staff, to start turning results around."
Ferdinand has made seven appearances for the West London outfit so far this season.