Javier Clemente's side started well but were unable to make the most of a couple of promising counter-attacks.
Gabon came back into the match as the first half progressed, but they were also wasteful with the ball in the final third.
However, with just six minutes played in the second half, Abdelsalam Omar lobbed the ball in from outside the area to give Libya the lead.
That was the way it stayed until Gabon introduced Daniel Cousin, who made an immediate impact with an equaliser from the penalty spot.
Both teams pushed for the winner in normal time and Cousin had a header saved, but extra time was required.
However, it remained 1-1 so the match went to penalties and Gabon missed their second kick.
Their opponents took advantage of that to move 4-2 ahead, and the win was confirmed when Gabon failed with their fourth penalty.