Town should have hit the front after just five minutes, when Branco headed a Massimo Luongo free kick over the crossbar.
It soon proved a costly miss as Kedwell fired the hosts ahead four minutes later, finding the corner of the net after Wes Foderingham came off his line.
The visitors hit back in timely fashion through Williams, who led a swift counter-attack before cutting inside his marker and firing beyond Bywater on 43 minutes.
Gillingham emerged from the break reinvigorated and moved ahead once more on 51 minutes, as Hause left Foderingham with no chance by rifling into the roof of the net from inside the area.
Mark Cooper's side struggled to restore parity but, just when it looked like Gillingham would hold on for their second win, disaster struck as Bywater diverted Branco's cross into his own goal to leave the hosts deflated.