The Swans defender was handed a second yellow card in the early stages of the match, which ended in a 1-0 victory for the Blues at the Liberty Stadium.
"If it's a minute, two minutes or five minutes into the game it doesn't matter," Terry told the club's official website. "I am speaking as a Chelsea player but when you sit back and look at it, it is two yellows.
"Fair play to the ref, it's a big decision to make early on and I thought he made the right one. He gave him a yellow before on the halfway line and that one just outside the box was probably even more of a yellow.
"Thankfully we managed to take advantage even though Swansea did well. If they continue to play like that, I'm sure they'll be okay."
Chelsea remain two points behind leaders Liverpool in the Premier League table.