Spurs have not won any of their three Premier League games so far, losing their opening match of the season before drawing their next two.
"As players, when you join a new club, sometimes it takes a while," said Defoe.
"It is not different for manager. When a new manager comes in, straight away you are going to be under pressure. It is a big club.
"But it takes time. It is still early days. It is important to give managers a chance. We have got a good squad at the club, a strong squad. I am sure we will be fine."
Villas-Boas replaced Harry Redknapp as the Tottenham boss in the summer.