The Reds prevailed 3-2 at the King Power Stadium in an entertaining Premier League affair that twice saw them open up a two-goal lead.
Leicester refused to give up, though, and after bringing it back to 3-2 Jamie Vardy squandered a glorious chance to level up from the penalty spot.
Klopp is unhappy with the penalty call, awarded after Simon Mignolet brought down Vardy in the box, but he felt that the decisions evened themselves out overall.
"It is not a time to be completely cool but overall I was happy with our performance," he told BBC Sport. "The boys deserved the three points. The free kick for the first goal was not a free kick. It was a foul on Simon [for the second goal].
"At the start of the second half we played the same football like Leicester - kinda hectic. Simon saved the penalty and then I think we really deserved the win.
"I really thought in a moment like this so many things are positive. The three points are the proof for the public that we are still here, everything is good. Let's make the next step together."
Prior to this evening's match in the East Midlands, Liverpool had lost a total of 20 points from winning positions since the start of last season.