The Black Cats beat Manchester United on penalties on Wednesday to set up a March 2 final with Manchester City.
However, Carragher believes that Gus Poyet's side, who are currently 19th in the Premier League, have enough quality in their side to avoid the drop.
"A team who have beaten Manchester City and Everton in the Premier League, and knocked Manchester United and Chelsea out of the Capital One Cup, clearly have the talent to stay up," Carragher said in his Daily Mail column.
"If they fail to do that, they will only have themselves to blame because they shouldn't have any other distraction.
"Sunderland fully deserve their first trip to Wembley since 1992 and they should enjoy every minute of the Capital One Cup final. In no way, though, should their progress in this competition be used as an excuse for the position they currently occupy in the Premier League table — and if they end up being relegated in May, they should not compare themselves to either Birmingham or Wigan."
Sunderland entertain Kidderminster Harriers in the FA Cup this afternoon. Catch live commentary of the game here.