Despite the speculation, Graham believes that the North London outfit will keep their faith in the ex-Chelsea boss if they qualify for the Champions League by the end of the season.
"It is going to take time," Graham told talkSPORT. "Bringing in half a team at the same time, it is going to take time for those players to settle in, and that is the problem they have got.
"I think they will stick by AVB. If they can pick up some points over the Christmas period and are still in touch with that fourth position, I think they will stay with him."
Spurs will take on Manchester United at White Hart Lane on Sunday.