James Collins then mis-kicked his backpass to allow Michu to double the lead moments later.
"We made two very poor mistakes by our two most experienced players on the field which cost us dearly," said Allardyce. "These mistakes were punished by Swansea.
"I'm disappointed because we got a good solid base last week and it looked apparent to me we were in control again. We were under no pressure but to make those mistakes, at this level, is unbelievable.
"We made things hard for ourselves. Our performance overall in the first half, apart from those two mistakes, was very good."
Next up for West Ham is a home tie against Crewe Alexandra in the Capital One Cup on Tuesday night.