Two goals in less than four minutes from Alexis Sanchez put Arsenal back in control after the hour mark, though, and the visitors held on for all three points despite James Tomkins giving Palace some late hope.
The result brings an end to Palace's club-record eight-match unbeaten streak in the Premier League to leave Roy Hodgson's side just one point clear of the relegation zone ahead of a New Year's Eve showdown with Manchester City.
Arsenal, meanwhile, keep pace with the top four after seeing Chelsea, Liverpool and Tottenham Hotspur all win earlier in the week, picking up only their third away victory of the season.
The match was Arsene Wenger's 810th as a manager in the Premier League - equalling Sir Alex Ferguson's all-time record - and his side were the first to threaten when Granit Xhaka flashed a 25-yard effort over the crossbar.
Long-range shots proved to be something of a theme in the opening exchanges as both sides struggled to break down the opposition, with Yohan Cabaye twice letting fly without troubling Petr Cech in the Arsenal goal.
The visitors soon began to gain an element of control over the game, though, and Jack Wilshere and Xhaka both squandered clear chances when failing to properly connect with the ball on the edge of the area.
Wilshere then drew a first save from Julian Speroni from range, but it was an even more unlikely source who broke the deadlock for Wenger's side when Mustafi converted a rebound at the back post after Speroni could only parry Alexandre Lacazette's wicked effort into the defender's path.
Arsenal almost made it two in the space of five minutes on the half-hour mark when Sanchez's defence-splitting pass found Ozil inside the area, but the German could not squeeze his finish past Speroni from close range.
Sanchez was the next to test Speroni with a bouncing long-range free kick, but Palace fired a reminder of the precarious nature of a one-goal lead moments later when Wilfried Zaha curled a low strike past the far post.
It was Arsenal who were very much on top as the first half wore on, though, and Lacazette flashed another shot wide of the near post before Ozil squandered another close-range chance, failing to tee up Hector Bellerin or Lacazette for a tap-in.
The Gunners would have gone into half time wondering how they were only a goal to the good, but they quickly picked up where they left off with a chance inside one minute of the restart as Bellerin failed to convert at the back post.
It looked as though Arsenal's profligacy would come back to haunt them when Palace levelled things up moments later, with former Tottenham Hotspur winger Townsend planting a firm first-time finish past Cech after being found by Zaha.
Suddenly Palace - who had hitherto barely threatened Cech's goal - were on top and began pushing for a quick second as both Ruben Loftus-Cheek and Jeffrey Schlupp wasted good openings from a tight angle.
Arsenal weathered the Palace storm, though, and they regained the lead shortly after the hour mark when Sanchez's powerful snapshot zipped past Speroni at the near post.
Having scored three goals in the space of less than five minutes during their last outing against Liverpool the Gunners once again quickly followed one goal with another as Sanchez doubled his personal tally moments later.
The Chilean's second was the pick of the bunch too as he plucked Wilshere's inch-perfect pass out of the air with his thigh before sliding his finish past Speroni.
Mesut Ozil and Bellerin both had chances to take the game beyond Palace once and for all, but Arsenal could not kill the game off and the home side's belief began to grow during the final 10 minutes.
Zaha looked the most likely to inspire a late comeback, but he squandered a clear chance when mistiming his header before Bakary Sako threatened twice in quick succession.
Palace did eventually pull a goal back with just a minute of normal time remaining when Tomkins rose highest from a corner to plant his header home, but it was too little too late for the hosts as Arsenal held on.
The Gunners stay sixth in the table, but move level on points with fifth-placed Tottenham having extended their unbeaten streak to seven matches, whereas Palace's longest unbeaten run since 1990 comes to an end.
Crystal Palace (4-4-2): Speroni; Kelly (Fosu-Mensah, 80'), Tomkins, Dann, Schlupp; Townsend, Cabaye (McArthur, 53'), Milivojevic, Loftus-Cheek (Sako, 70'); Benteke, Zaha
Arsenal (3-4-3): Cech; Chambers, Mustafi, Koscielny; Bellerin, Xhaka (Maitland-Niles, 87'), Wilshere, Kolasinac; Ozil, Lacazette (Coquelin, 75'), Sanchez