The national side's manager used the phrase "feed the monkey" during his half-time team talk while encouraging to team to provide more service to Andros Townsend.
"How can anybody be offended by that? It's not racist. Feed the monkey is the same as saying 'feed the goat' [to former Manchester City striker Shaun Goater]," the Daily Mail quotes Barnes as saying.
"The most important thing is the element of intention. Does the person mean to say something that could be considered to be prejudicial, racist or discriminatory and would the person it is directed at be offended.
"No-one can possibly be offended by these comments. If you told the joke to Wayne Rooney, would he be offended? I don't think so. You cannot be offended by that – which player is offended? Let's hear from them and let them tell us why they are offended."
Anti-racism group Kick It Out has called for the FA to investigate the comments.