However, Reims equalised a couple of minutes later through Gaetan Charbonnier, who found the top corner of the net with a neat finish.
The lead was restored five minutes before the break as Dijon controlled play, when substitute Jeremy Bela connected with a cross at the far post to give Dijon a 2-1 lead.
Dijon nearly scored a third on the stroke of half time, as Agassa fumbled the ball into the path of Johan Gastien but the Reims goalkeeper made up for his mistake to save the shot.
Agassa kept Reims in the tie after the restart, as Bela's curling free kick was matched by the Togolese goalkeeper.
Bela continued to threaten the visitors' goal throughout the second 45 minutes, with each effort inches away from a two-goal cushion for the hosts.
Despite not finding a third goal with a handful of chances, Dijon did enough on the night to secure a 2-1 victory over Reims.