The Spanish side reached the quarter-finals of the Champions League last season, but they are currently struggling in La Liga, with 14 points leaving them in 15th position in the table.
"The results aren't helping us follow a line. We have good first haves but then we're not being consistent," he told reporters at a press conference.
"We're conceding lots of goals and I'm the main one to blame for that. In terms of morale, the team's not in a good place for having let in those three goals.
"The problem could be down to lots of issues - but it's not a physical thing, perhaps it's that we're not defending as well as we should, and we certainly can't put anything down to the younger guys."
Caballero has been linked with a move to Manchester City.