The Red Devils twice held a two-goal lead at the King Power Stadium this afternoon, but Nigel Pearson's charges staged a remarkable comeback to inflict a second league loss of the season on their opponents.
"How is it possible to give this game away? We have given the game away," Van Gaal told Sky Sports News. "Leicester has proven already in four games that they have a strong character and that when they are 3-1 down, you can play all the game with more possession but you cannot do these kind of things."
To make matters worse for United, defender Tyler Blackett was shown a red card late on, although Van Gaal is confident in the ability of his individual defenders, despite them having conceded eight times already this campaign.
"It is not pleasant when players are injured or sent off but we can manage," he continued. "I don't think it is the weakness of our defence, it is the weakness of our defensive organisation from the team. That we have to do better. You have to know when you can pass the ball and when you have to get rid of the ball.
"In this game, at 3-1 I felt we had played fantastic football but then you cannot make the errors we did. We have to continue and improve. We created a lot of chances, we made fantastic goals so that is a plus but we lost the game and that is the most important thing."
Next up for the Red Devils is a home match against West Ham United next weekend.