Charlie Mulgrew's 25th-minute penalty gave the visitors the interval advantage but there was no surprise when the hosts levelled in the 51st minute, midfielder Dor Peretz making up for conceding the penalty with a barge on Steven Naismith with a close-range finish.
In the 61st minute Scotland defender John Souttar was sent off for picking up the second of two yellow cards before defender Kieran Tierney turned a cross from Beram Kayal past Allan McGregor with 15 minutes remaining to complete Israel's comeback.
Scotland, who beat Albania 2-0 in their section opener at Hampden Park last month, have to face the Albanians away and Israel at home in their final double-header next month.
A defiant McLeish, whose side first play Portugal in a friendly at Hampden on Sunday, said: "We had a great performance against Albania and a good win.
"We have to pick it up, simple as that. That's what it's like in football, you lose a game and you go on to the next one and do everything we can to win
"We are obviously going to get a bit of stick for this and it's up to us to bounce back, that's what I would normally expect from Scottish people.
"Whatever it takes (to qualify). I will look to win both games but whatever it takes.
"I've had a few setbacks in my career but also a few probably more really good moments and that's what we have to focus on. It was difficult in the second half.
"Probably the goals we lost were probably harder to go in than one or two chances they had.
"We were under the cosh admittedly we will look at the game and analyse it and try to put it right."
Scotland were booed off at the end of the game by the Tartan Army but McLeish said: "The fans want to win. I'm right behind the fans. Obviously they give the players energy if they support them and I urge the fans to support the players."
McLeish felt Souttar was hard done by after being sent-off for two fouls on Israel striker Munas Dabbur.
He said: "I thought it was soft, very, very soft. The ref was 60 yards away, he wasn't up with play.
"Maybe he got something in his ear. Maybe John will say he should have played it quicker.
"We were going to take John off it was really ill timing for us. Unfortunately play stopped for a free-kick for them and a red card for John Souttar.
"We had a long way to go playing with 10, it's never easy 10 against 11 and Israel had one or two players in good form.
"They deserved it on the night although we're disappointed with the goals we lost."
Andreas Herzog won his first game as Israel boss and ahead of his side's Group C1 game against Albania on Sunday, he said: "At the end we had a lot of chances and it was a deserved win.
"We had a really good day and it was not the best for Scotland. A 2-1 win is a great result for us."