The PFA Player of the Year has been the subject of reported interest from Real Madrid for several months after a superb season that saw him score 21 goals in the Premier League.
Neville believes that should Bale leave White Hart Lane, it will leave Andre Villas-Boas's side significantly weaker.
"It goes without saying if Tottenham lost Bale, that would be a massive blow," The Mirror quotes Neville as saying. "But it looks at the moment, there are no noises coming out to suggest he is wanting to leave or that Tottenham are willing to sell.
"I think this year could be tougher for Tottenham to get into the top four because Arsenal will improve greatly from where they were last year. I think you will see Arsenal sign two or three players so Tottenham will have to do the same to compete."
Bale also won the Football Writers' award last season, despite Spurs finishing fifth in the Premier League.