Saints quarterback Drew Brees threw four touchdown passes against a Panthers defence that had not allowed more than two in a game this season as New Orleans ended Carolina's eight-match winning streak.
Rivera hinted that he thought the Saints benefited from home advantage during the match, a theory he is keen to test when the two sides meet again in Carolina in a fortnight's time.
"I'm surprised and disappointed. We're a better team than we played today. They have the crowd behind them and they're on turf, not natural grass, which is a little bit different," Rivera told The AP.
"It helps them as far as their speed and their timing. The things that they do here, they do very well. We'll see how things go when they come to Carolina."
The result means that the Saints now have sole control of the NFC South with just three games of the regular season remaining.