The spinner finished his career in Australia after England had lost the first three Tests of the series, and he has hit back at suggestions from Gooch that his poor form was the main factor in his choice.
"I never spoke to Goochy about it," Swann, who was struggling with a long-standing elbow injury before retiring, told reporters. "I'm surprised actually, I thought he would have talked to me. I said at the time, I don't think it's right and proper to carry on in the sport when you're a hindrance to the team.
"It was an injury issue. Some people think differently, some people think you should just carry on regardless, but I don't and I never will.
"He's a bit misinformed thinking that I just left because of form. It wasn't because of form, it was because I couldn't turn a cricket ball which, as a spin bowler, you are then useless to your team."
Swann had taken seven wickets in the first three Tests of the Ashes series.