The centre-back was sent off in the 25th minute after fouling Stewart Downing as he burst clear.
Sherwood told Sky Sports News: "The game changed on the red card. I thought before that we were comfortable and created the best chance of the game when Emmanuel Adebayor had a chance.
"It was always going to be a tough place to come, and when we go down to 10 men and they score a minute after it is going to be a difficult afternoon.
"Then, obviously, the free-kick goes straight through the middle of our wall, which isn't ideal, and then you've got the mountain to climb."
Tottenham face Aston Villa in their next league game on Sunday.