Neil Adams has been appointed as the club's new manager, but Roberts believes that the former Under-18s coach has been left with little time to make an impact.
"I think the timing is very, very poor," Roberts told talkSPORT. "There are only five games to go and what do you expect Neil to do in that space of time with the games coming up?
"There was a period from mid-December to mid-January when they didn't win in eight games. They weren't playing well, they weren't scoring goals, and I think the crowd were calling for Chris to go then.
"I think that probably would have been the right time [for Hughton to go] because you give the new man enough time then to come in and have an impact and the transfer window to bring his own players. With five games to go, it is a bit of a panic situation."
Adams's first match in charge will be at Fulham on Saturday.