It did not take Livingston long to get back on level terms as Myles Hippolyte was left in space 20 yards out, giving him time to blast a dive into the roof of the net.
Hibernian were ahead again less than a minute later though when Jason Cummings played his way in from out wide before guiding the ball into the bottom right.
Livingston's hopes of finding a second equaliser were dealt a blow just before the break when Darren Cole picked up his second booking of the night, leaving the hosts a man short for the second half.
The visitors had a chance to extend their lead shortly after the restart when Kyle Jacobs brought McGeouch down in the box, but the on-loan winger could not convert from the spot after winning the penalty.
Lewis Stevenson quickly got the ball back for Hibernian, but his shot from outside the box was blocked by Jordan White, preventing the visitors from increasing their advantage.
Hibs forced a string of corners as they continued to pepper the Livingston box, but Paul Hanlon fired high from a rebound, while Fontaine headed wide from the edge of the six-yard box.
The result keeps Hibs third in the table, just a point adrift of second-placed Rangers, while Livingston remain bottom.