The Black Cats are yet to pick up their first league win of the campaign, and the American believes that their forthcoming matches against Stoke City and Swansea City will be crucial.
"We need to be tougher. It is too easy sometimes. All of us - strikers, everyone - have to be nastier; make sure, when people play us, they have to really work for it," The Mirror quotes Altidore as saying.
"We have played fairly well, but now we have to get the first win. We don't want the shadows of last year following us around this season.
"We have had a decent start and now, with the home schedule we have, we have to take three points. There is room for improvement. We have to play better."
Sunderland lie in 17th place in the table with four draws from their first five matches.