Roberto Soldado scored the only goal of the game from the penalty spot in the 78th minute to take the North London side fourth in the Premier League, but the Portuguese coach was unhappy with tension from the crowd.
"We looked like the away team. We played in a difficult atmosphere with almost no support," said Villas-Boas. "We have a wonderful set of fans but they can do better. We don't need the negativity of today.
"Away from home their support has been amazing; we play with no fear and we need that atmosphere at White Hart Lane. We didn't have the support we should have done. There was much anxiety from the stands, the players had to do it alone.
"We spoke about it at half-time. I told the players that we would have to do it on our own. They had to dig deep and look for the strength within themselves. They also believed that it's not easy to play in this stadium when the atmosphere is like this."
Spurs' next home match sees them take on Sheriff Tiraspol in the Europa League on November 7.