Substitute Eduardo Vargas scored two late goals to have seemingly secured a draw for the Hoops, but Steven Caulker turned the ball into his own net deep into stoppage time to hand all three points to the visiting Merseysiders.
Speaking after the match, Redknapp conceded that his players were "naive" as they twice allowed Liverpool to score on the counter-attack late on, but he also stressed his happiness at their overall performance.
"That's the best I've seen the team play since I've been here. We were naive. You take the point today because, in our situation, it gives us all a big lift," he told Sky Sports News.
"We kept coming back. I just felt we were fantastic today. We were solid, but naivety cost us."
The result has left Redknapp's men bottom of the Premier League table with four points from seven games.