Chiles was talking about Tuesday's match between England and Poland when he mentioned that there would be 15,000 Poles at Wembley for the match.
Analyst Lee Dixon then expressed his desire to see those fans "crying" by the end of the game, to which Chiles responded: "I'm trying to get some building work done at the moment actually, quite seriously. Be careful."
The 46-year-old, whose mother is from Croatia, insists that he meant no offence by the comment and that it was a genuine remark aimed at the Polish builders who are currently working on his house.
"I made the remark in relation to the nice Polish builders currently working at my place, who I knew would already be fed up having watched their team lose to Ukraine," Chiles said in a statement.
"No offence was intended - apart from anything else I could hardly be prouder of my own East European background. But to anyone who thought what I said constituted some kind of lazy stereotyping, and was offended by it, I certainly apologise."
England will guarantee their place at the 2014 World Cup with a win over Poland on Tuesday.