It was reported on Monday that Sherwood would be relieved of his duties this summer despite having another 12 months on his contract at White Hart Lane.
"Franco Baldini and Daniel Levy have to take some responsibility because which players have really come in after the sale of Gareth Bale and made an impact? Maybe you could say Christian Eriksen but apart from that there haven't been too many," he told Sky Sports News.
"That's what I find hard to understand. If Tim Sherwood does leave at the end of the season, how can a director of football like Franco Baldini keep his job? I find that amazing.
"The young managers get the sack and the people that bring the players in don't have any responsibility. I find that staggering in this day and age."
Tottenham continue their Premier League campaign this evening (Monday) when they entertain Sunderland.