The two clubs are currently in talks over a reported world record £105m deal for the winger, who is said to be keen on becoming a Madrid player this summer.
"You never know but when the player is willing to leave, it is not easy to keep him when he wants to go," Calderon told Sky Sports News.
"Spurs don't want the player to go, but it seems that the player has decided to leave, so it will be difficult. The figure being talked about is a huge amount of money."
Some reports have claimed that the clubs are discussing which Madrid player will move to White Hart Lane as part of the deal.