The Villans have had an initial £9m offer rejected for the former Newcastle man, who Whelan insists is the best left-winger in the Premier League and is reluctant to lose.
"It's £10m or there is no deal," he told The Mirror. "They offered us £9m and we refused it; firstly because we would like to keep Charles, and secondly because if he is to go, then we have to get the right price for him.
"Charles has played fantastically well in the last year and, with his pace and penetration, there's no doubt in my mind that he is the finest left-winger in the Premier League."
N'Zogbia, who is entering the final year of his contract, has refused to sign a new agreement with the club, which means he would be free to leave the DW Stadium next season.
"We owe him a lot, not least for his goal against West Ham that helped us stay up," Whelan continued.
"If the right offer comes in, we would consider it, if it was in his best interests - but it has to be the right offer."
N'Zogbia has made 86 appearances for Wigan in two and a half years at the club, scoring 15 goals.