Posh led the game twice, with goals from Lee Tomlin and Nathaniel Mendez-Laing coming either side of a Glenn Murray penalty.
Posh had the game's first chance, as a defensive error by Danny Gabbidon gave Gayle the opportunity to open the scoring, but he was denied by a good save from Julian Speroni.
Bolasie was Palace's main creative force in the first half, and another cross from the winger fell to top scorer Glenn Murray, but the striker somehow contrived to head over from four yards out.
Just before the half-hour mark, Lee Tomlin gave Posh the lead. The centre-forward took on three defenders before firing a low shot into the corner.
Peterborough continued to attack in search of a second goal, and minutes after being booked, Damien Delaney gave the ball away to Mendez-Laing and his cross just evaded Tomlin and Gayle.
However, Palace got back into the game on the stroke of half time, when Danny Gabbidon was brought down in the box following Garvan's corner. Glenn Murray stepped up to level the scores and notch his 30th goal of the season.
Palace started the second half brightly, and England international Wilfried Zaha ran at the Posh backline and his cutback was met by Murray, whose effort was cleared off the line by visiting captain Tommy Rowe.
Peterborugh regained the lead with 25 minutes remaining, as Mendez-Laing stepped in from the right and curled a shot past Speroni with his left foot.
However, with just three minutes left on the clock, substitute Kevin Phillips picked the ball up just inside the area and smashed it into the top corner.
As the game moved into extra time, Mile Jedanik got the winner for Palace to send Posh down.