Poyet also said that the goalkeepers proved to be the difference in the contest, with Birmingham's equaliser coming from a mistake by Tomasz Kuszczak.
"The best thing that occurred early on was to concede a goal," PA Sport quotes Poyet as saying. "We were sloppy and not playing at the tempo I like.
"After that goal we were a good team to watch and we created problems until the 92nd minute when Birmingham equalised. The last four minutes contained the mistake by my goalkeeper and a brilliant save from Jack Butland.
"We dropped two points against Birmingham and we only have ourselves to blame."
Brighton are eighth in the Championship, five points behind the playoff places.