The former England boss has picked up just one point from his three league games since taking over from Alan Pardew at Selhurst Park, and also saw the Eagles held to a goalless draw by former club Bolton Wanderers in the FA Cup last weekend.
Allardyce will return to another one of his old clubs on Saturday when Palace visit West Ham United, and he admitted that it is vital for his side to win having missed the opportunity to pick up three points in his very first game at the helm, when the Eagles drew 1-1 with Watford in a match that saw Christian Benteke miss a penalty.
"[Getting a win] is massively important. Deep down inside, even though I didn't say it, reflecting on the Watford game it was bitterly disappointing and how costly that draw was," he told reporters.
"The victory we should have gained in that game, with it being my first game would have made everyone a bit more comfortable. Yes we got a good performance, yes we should have won but we didn't and of course it was very costly.
"Missing the penalty when we did and now we see we have gone four games without a victory so we see how important it was to clinch that game. We just have to make sure that we get it sooner rather than later because the longer it goes on, the more fearful players can get so we need to put that away as soon as possible."
Palace currently sit 17th in the Premier League table, just one point above the relegation zone.