The Canaries are currently two points above the relegation zone following four defeats in their last seven games.
"There might be times you do think about it but we just don't know," he told the Pink Un. "All your efforts and what you do on a day-to-day basis must be focused on the games.
"I don't know what it will take, but I am still confident and you have to be. It is important that no one here panics."
Norwich return to Premier League action on Saturday when they host title contenders Manchester City.