The away side dictated play for much of the opening stages and opened the scoring midway through the first half.
A corner from debutant Campana was cleared only to Moxey, who lashed the ball into the corner of Stuart Nelson's goal from 25 yards.
Holloway made 11 changes to his team at halftime and the newly promoted Premier League side doubled their lead with little more than 20 minutes remaining.
Dobbie picked the ball up inside the Gillingham half and drove at the defence before curled a low shot beyond the reach of Nelson.
Dobbie was also involved in Palace's third goal, laying a through-ball into the path of former Arsenal striker Owusu-Abeyie, who finished casually.
Bolasie almost furthered the Eagles' lead with 15 minutes remaining, but his effort cannoned back off the crossbar as Palace dominated from start to finish.