The former Fulham boss said that he would prefer it if his key players were not called up to represent Team GB.
"Ideally I wouldn't want any of our players involved," the Wales boss told BBC Sport. "I can't say I'm delighted about the fact that if they get to the final, four days later we've got a game.
"But there's nothing we can do about it. We've just got to get on with it."
Wales take on Bosnia-Herzegovina in a friendly on August 15, four days after the Olympics football final will take place at Wembley.
British Olympic Association chief executive Andy Hunt previously suggested that Wales stars Craig Bellamy and Gareth Bale have agreed to take part in the Team GB initiative, and are on a shortlist of 80 players.
Coleman said that he understands why his players would wish to take part in the Olympics, but added that he is unaware of any agreement that grants them clearance.
He added: "Most of the clubs I talk to, or managers that I talk to, are not that happy about it.
"Obviously the clubs they've talked to have agreed to it otherwise they wouldn't come out with that statement."
The draw for the tournament will take place on April 24.