The 29-year-old sustained a dislocated collarbone in a collision with Orica-GreenEdge's Simon Gerrans 200m from the finishing line in Harrogate yesterday.
Despite boarding the Omega Pharma-Quick Step team bus this morning, Cavendish has since been ruled out of the remainder of the race by the team doctor.
"I'm absolutely devastated," Cavendish told BBC Sport. "I had some optimism that the swelling would go down overnight, but it's worse this morning.
"I knew straight away because normally in crashes I bounce back straight away.
"This was the first time in my career that I knew something was up, but I wanted to finish. I was able to do that, but I was in pain."
Cavendish has won 25 Tour de France stages and was voted the competition's best sprinter of all time by French newspaper L'Equipe in 2012.