The Northern Irishman ended a four-year reign at Villa Park by resigning just four days prior to the start of the 2010-11 season.
The now-Sunderland manager achieved back-to-back sixth-placed finishes while at the Midlands club. He also spent heavily and amassed a squad wage bill of £80m.
However, McLeish thinks that the 60-year-old's departure is still impacting the club nearly two years on.
When asked if he was still picking up the pieces following O'Neill's exit, McLeish replied: "Yes, definitely. There are a lot of quality players who have left the club since Martin left. He had a tremendous time at this club and did a great job getting them to heights it is difficult to follow.
"Especially given the loss of a lot of the players Martin signed and not replacing them. That makes it tough [first] for Gerard and for me. But I have got to believe I can get Villa back to the heights where fans expect them to be.
"I know where the club is at the moment and I took the job under those circumstances because I see better things in the future."
O'Neill will return to Villa Park this afternoon as his Black Cats side face Villa in the Premier League.