The home side looked sharp in the opening exchanges despite bringing in 11 new faces, and Dwight Gayle went close with a low drive after just three minutes.
© Getty Images
Newcastle began to find joy in wide areas, but couldn't find a leveller until the 36th minute, when Emmanuel Riviere rifled a finish beyond Wayne Hennessey.
Newcastle were given the chance to put themselves ahead for the first time in the tie two minutes after the break when Paddy McCarthy felled Sammy Ameobi, and Riviere made no mistake from the spot to net his second goal of the evening.
Gayle came close to levelling 15 minutes from time, but his header shaved the outside of the post.
Newcastle had plenty of chances to put the game to bed, but Hennessey denied Adam Armstrong a debut goal shortly after Riviere passed up the chance to secure a hat-trick when he placed a 25-yard shot narrowly over the bar, and the miss proved costly.
In the 91st minute, 19-year-old Sullay Kaikai - who had only been on the pitch for 10 minutes - reacted quickly after his initial shot was blocked and netted his first Palace goal with the rebound to send the game into extra time.
However, Newcastle dug deep and scored the winner in the 112th minute when Paul Dummett headed home.
There was still time for Palace to go mightily close to a very late equaliser, but Elliot produced a flying save to deny Adrian Mariappa.
Newcastle will travel to Manchester City in the fourth round.