The Portuguese endured a torrid time during his reign at the Blues, lasting just 10 months at the club before being sacked.
Villas-Boas, who returns to Chelsea as Spurs boss with both teams needing a win to help secure Champions League, is unsure as to what kind of reception he will receive from his former supporters after witnessing the heckling aimed at Benitez.
"He made public is frustrations and that made it easier for him to work more comfortably, as it was affecting the team's balance," Villas-Boas told ESPN. "For a manager to get past the frustrations of the fans is difficult, but the fans represent the passion and emotion in football and they are entitled to their opinions.
"For me, [the return to Stamford Bridge] is not very special. It is a club I have gone past and I have good and bad memories. It was a period that did not finish the way that I would have liked.
"Before I was manager there, I was in Jose's [Mourinho] team and I had some wonderful years there. Probably the ones that I spent there as a manager aren't that special."
Spurs travel to Chelsea on Wednesday night.