Gabriel Obertan scored the only goal of the game to fire the Magpies to victory at St James' Park on Saturday and ease the pressure on manager Alan Pardew following the club's poor start to the campaign.
Colback told the club's official website: "It's a massive relief. It's been a tough start for us and everyone involved in the club. It's been difficult but we've always had that belief.
"We've worked hard on the training field and it wasn't the prettiest of wins but that doesn't matter at this stage - we've got the win and a good footing to push on now and start playing well and winning more games.
"The last 20 minutes felt like a long time, but I think when you haven't won in a while it's sometimes difficult when you get 1-0 up - which is the first time we've scored first this season - as the margins are fine, and they had chances and they could have equalised. But maybe that's our luck just turning a bit and we managed to hold on."
Newcastle remain in the relegation zone despite their win over the Foxes, with seven points from eight games.