The 27-year-old Olympian, who came under fire three years ago when a photo emerged of him with a bong, was seen by Crowder three weeks prior to the incident in a club with the rapper.
"I was on South Beach probably three weeks before [the photo came out]," Crowder said on WQAM. "Everybody knows who Young Jeezy is. He was in [the club]. It was crazy because it was Young Jeezy and Michael in the next section over from us.
"I shook his hand and he was blowing people off because he was drunk too and I see him hit the blunt later on when Jeezy passed it. No lie! I don't get star struck or impressed by many people but when I saw Young Jeezy hand Michael Phelps a blunt, that was impressive to me."
Crowder also described Olympic swimmer Ryan Lochte a "hippie", who smoked weed "all the time" during his time at the University of Florida.