Henri Lansbury came close to opening the scoring inside the opening 10 minutes when he struck the crossbar directly from the corner, before a 25-yard effort from the midfielder went just over a few minutes later.
The home side survived the early spell of dominance from their Championship opponents before creating a chance themselves towards the end of the first half, but Karl Darlow put Eli Bako's shot behind.
Courey Grantham also tested the Forest keeper shortly before the break, but Darlow was able to gather from his finish.
Dorus de Vries took over between the posts in the second half and was called into action just a couple of minutes in when Laurie Wilson tried to drive the ball in from close range, before James Reid headed into the bar from the resulting corner.
However, Cox was left unmarked in the box just a few moments later, and was able to head in from a Danny Fox cross to break the deadlock.
The goal seemed to boost Forest's confidence as they took control of the match, adding a second just four minutes later when Ben Osborn calmly slotted the ball in from out wide.
Cox completed his brace within six minutes, chipping the ball over Smith as the keeper came off his line to try to deny the striker.
Osborn also grabbed a second just before the end of the game, heading in from a Lars Veldwijk volley from close range.