The Black Cats were knocked out of the domestic competition by Middlesbrough last night and O'Shea believes that he and his teammates need to step up.
"Chances to get into the quarter-finals of a cup don't come around every week and we've let ourselves down," O'Shea told the club's official website.
"We need to drastically improve on this very quickly because we have a massive game now on Saturday against Aston Villa. We have to improve on a lot of things and it will start as soon as the lads come into training."
Sunderland currently sit 14th in the league table, just five points above the relegation zone.