Two goals in the space of eight second-half minutes handed the Bees the win following an evenly-matched opening 45.
"We were really bright in the first half and were the better team. I was delighted with the way we played up to half time," McCarthy told the Ipswich website.
"It was an appalling goal to give away for the first one though and that seemed to drain the confidence out of the side. What worried me what how we unravelled after the first goal. You want to growl about that but make sure you look safe and secure after it but we didn't look any of that. We looked like we would concede another goal and that is what happened. That concerns me.
"That first goal had a real adverse effect on us. I don't know why. It gave them confidence and apart from a couple of half chances, they looked fairly comfortable after their goals."
The result came just four days after Town crashed out of the EFL Cup at home to League Two side Stevenage.