Romain Hamouma tried to level for the visitors just two minutes later, but his shot from a tight angle was just kept out.
The home side doubled their lead at the start of the second half through a Franck Signorino strike after a quick break from Charbonnier.
Hamouma and Benjamin Corgnet both had chances to level shortly after, but their finishes failed to find the target.
Saint-Etienne continued to press, and Corgnet almost got them back on level terms just after the hour mark, but his header from a Francois Clerc cross was kept out by the crossbar.
Clerc continued to create chances for the visitors, heading the ball on for Brandao a couple of minutes later, but Agassa made the save.
Saint-Etienne still looked the more likely as full time approached, and eventually got the breakthrough in stoppage time when Clerc blasted an effort in from the right-hand side of the area.
The result leaves Saint-Etienne fourth in the table, while Reims remain seventh.