Chris O'Grady went close for the Seagulls on 22 minutes when he flashed an effort across the face of the goal, but the effort was well saved by Stuart Nelson.
Dale Stephens evened things up before the hour mark with a smart finish from close range.
The South coast side then took the lead when Chike Kandi exchanged passes with Stephen Ward before slotting home on 66 minutes.
Rohan Ince was causing Gillingham problems throughout the second half, and he went close on 69 minutes when he burst into the box before firing wide.
Gillingham had the ball in the net inside the final five minutes, but the effort was ruled out for a push.