Nigel Doughty has also stepped down as chairman, but will remain the club's owner under Financial Fair Play rules which are expected to be introduced to the Championship next season.
A club statement read: "Nottingham Forest can confirm that Steve McClaren has resigned from his role as manager.
"He has left the club without compensation - nor will he be seeking any.
"The club can confirm that the search for a new manager will begin immediately."
Forest have endured a poor start to the season, picking up eight points from their first ten games, and the side lost again on Sunday after they went down 3-1 to Birmingham City.
The former England manager was appointed over the summer after spells in the Dutch Eredivisie and German Bundesliga.
The East Midlands side travel to Coventry City in two weeks, after the international break.