The Ivory Coast international was sent off in the 39th minute of the game at the Liberty Stadium for hacking down Saints defender Maya Yoshida from behind to earn his second bookable offence.
Monk's side nearly held out for a point, but Victor Wanyama scored with 10 minutes remaining to hand Ronald Koeman's men the win.
He told BBC Sport: "I would question the first free kick against Wilfried Bony. Whether it was a yellow or not I don't know. The second one was a foul but there was no malice in it. It puts us in a difficult position.
"We were by far the better team in the first half. And in the second, the only shot they really had on goal was the one they scored from, which was the one time we switched off and got punished. But we will pick ourselves up and go again."
Swansea are now fifth in the Premier League following their second defeat of the season.