Downing, who arrived at Upton Park in a £5m deal from Liverpool in August, will face former club Aston Villa later today.
The 29-year-old, who has been capped 34 times by England at senior international level, has made nine first-team appearances this season without scoring.
His performances have been criticised by some Hammers supporters, as well as in the media. However, Allardyce is adamant that Downing is just going through the settling-in process.
"Other people's opinion is of no interest to me," Allardyce told reporters. "My opinion is the only opinion that matters between me and Stewart Downing.
"Stewart Downing had a difficult pre-season where he wasn't sure where he was coming or going. Finally he made a decision on West Ham United which we were delighted with and there's very good progression and much more contribution as the more games he plays.
"The more and more quality we see on a much more regular basis. There's a settling-in period like anybody else. People divide their opinion far too quickly in this game. Real Madrid have done it with Gareth Bale, haven't they?"
Downing scored 11 goals during 81 appearances for Villa.