Rutherford had trials with the club when he was young but he instead focused on athletics, which ultimately resulted in his gold medal in London.
"I was all right at football and I wasn't a bad player," he told the Sunday Mercury. "However I was a bit lazy to be honest and didn't work hard enough at it.
"I have been on Soccer AM since I won the gold. I skied my shot at the end of the show though.
"It's nice to be back in Birmingham for the Diamond League and to be back competing as I love being up here."
As well as his London gold, Rutherford claimed silver in the 2006 European championships.