The Blues, who currently sit top of the Premier League table due to an unbeaten run, will come up against Villas-Boas for the first time since he was sacked from Stamford Bridge following a nine-month stint.
"We have some very tough games and they will be a great challenge for us," Sky Sports News quotes Ivanovic as saying. "At the moment it's OK but the next two games will be very, very tough.
"It's always very difficult to play the game after the international break, but especially when it's big opponents like Tottenham away.
"Our games against Tottenham were very interesting and tough for both sides even before Andre went there. Now it is especially so for both sides with our ex-manager."
Tottenham sit fifth in the table.