The comments allegedly made by Sterling were to his girlfriend on an audio recording, with the Clippers owner stating that he would prefer it if she did not bring black people to games.
Sterling's alleged remarks have caused controversy in the NBA, and Rivers is convinced that it was the owner's voice on the recording.
"Yeah, I believe he said those things. I still want to make sure it hasn't been doctored, but yes is the answer," the Clippers coach told reporters.
"As far as believing those things, I heard what he said. Until someone tells me differently, you usually listen to what people say. I haven't given him his due process.
"The fans are in a dilemma as well. We want them to cheer for their players and their team because it's still their team. From what I got from the fans that I talked to, that's how they feel: 'This is my team and these are my players that I'm cheering for, and that's not going to change,' and I hope that continues."
The NBA have launched an investigation into the incident, and it is believed that Sterling's role in the league could be under threat.