The Swans endured a goalless draw in the second leg of their League Cup semi-final clash against the Blues at the Liberty Stadium, but still progressed to the final at Wembley, having picking up a 2-0 win in the first leg at Stamford Bridge.
"Demba Ba told me the ball boy held on to it," said Williams. "I saw him kick him in the rib and you can't do that to a young boy.
"At times tonight it was like a bar fight especially when they were pushing on.
"The team defended brilliantly. That's what we worked on. We wanted to bring them back here and score a goal and keep a clean sheet. We didn't score but we kept a clean sheet."
Swansea will face Bradford City in the final of the competition next month.