The Black Cats, who secured their Premier League status last term thanks to a late positive run of form, are still without a victory from their five top-flight matches this campaign.
Speaking after last night's 2-1 League Cup defeat at the hands of Stoke City, Altidore conceded that Gus Poyet's men are currently losing encounters that they should be winning, which if it were to continue, would result in the team fighting for survival again.
"I didn't think we deserved to lose. I don't think we played that bad, but this has haunted us the whole of last season and a bit this year. Games we should win, we're not winning," the USA international told the Sunderland Journal.
"It's something that we need to address now or else it's inevitable that what happened last season - and the pressure - will come back."
At the weekend Sunderland will play host to Swansea City.