The newly-promoted Premier League side are due to find out next month if they have been successful in acquiring the stadium that hosted the London 2012 Games.
"If they keep it for athletics, the only time it will ever get a big crowd again after the Olympics is when they hold the World Championships there in five years' time," Allardyce told The Times.
"You only have to look around the world at some of the Olympic Stadiums that have been built. They're white elephants now. There are weeds growing there. Look at the 'Bird's Nest' in Beijing. Look at Athens.
"The Olympic Park is a fantastic place, but it can't be left to rack and ruin. If a club with the history and fanbase and potential of West Ham don't go there, the concern would be that the park is left empty the vast majority of the time."
West Ham are facing competition from League One side Leyton Orient.