The Bosnian led the African side to the last 16 of the World Cup, where they suffered a 2-1 defeat against Germany.
The 61-year-old took charge of Algeria in 2011, but he has decided to step down following their exit from the tournament.
"Today, after living in Algeria for three years, my family obligations and the lure of new sporting challenges weighed heavily on my decision," he told the Algerian Football Association's official website.
"I leave proud of my record and in accordance with the FAF (Algerian Football Association)."
Algeria made it out of the group stage for the first time in their history at this year's World Cup.