The visitors were leading 30-17 with 10 minutes to go, but a late penalty try gave the Springboks a 31-30 victory.
The coach told Sky Sports News: "It was probably [the toughest defeat] of my whole coaching career. It's tough. We played pretty well in the first half and put ourselves under pressure with the two yellow cards but gave us the opportunity.
"We're gutted and we didn't deserve that final result. I thought the referee had a good game and did a good job. In the last 10 minutes they were throwing everything at us and we let Hendricks go through and the penalty try was unfortunate because we got narrow and allowed the winger to get outside us.
"There was a big improvement on last week. We put South Africa under a lot of pressure and scored some good tries. I'm proud of the effort and performance but gutted about the result."
Wales have never won an international match in South Africa.