The Blues have won just two of their last 15 Championship games and currently sit 17th in the table, nine points above the bottom three.
McCarthy has come under criticism by some supporters this season, but the 58-year-old was eager to remind fans of the impact he has had at the club since his appointment in 2012.
He told the Ipswich Star: "Some of the stuff about 'Mick McCarthy you've ruined our club', that's the biggest load of cobblers I have ever heard.
"I saved the club from relegation and I have had three top-10 finishes. The one time I have a bad season then people have turned their backs on me which is a bit of a surprise and it's disappointing.
"But hey-ho, I am not walking away. I have a year's contract to run and unless something else is done, no, I am not going, no way.
"I am looking forward to getting enough points to make sure we stay in this league. I will go and have a break, and I will come back full of vim and vigour again."
McCarthy replaced Paul Jewell in November 2012 with Town five points adrift at the bottom of the Championship and guided them to a 14th-place finish.