The 54-year-old has recently been linked with a return to the Republic of Ireland, whom he previously managed between 1996 and 2002.
McCarthy himself has refused to discuss the speculation, but centre-back Smith insists that if he was to depart Portman Road, then the team would end up having to start "all over again".
"We've just got to get on with our jobs. If it happens, it happens," the East Anglian Daily Times quotes Smith as saying. "We don't want him to go though because he's been really good for us.
"He's now had the chance to build his own squad, the performances have been good and there is a feeling we've got something to build on. A change of manager means starting all over again so we really want him to stay."
McCarthy is currently the favourite with the bookmakers to fill the Ireland void.