The Bees saw two penalty appeals turned down by referee James Linington en route to a 3-1 defeat at the hands of their in-form visitors, while the free kick which led to Cardiff's third goal was also contentious.
Smith acknowledged that his side should have dealt with Cardiff's physicality better, but believes that the officials were also at fault at Griffin Park.
"I thought the ref was inconsistent today. You've got to deal with the physicality that Cardiff bring. We've seen their team today - they're big and strong and robust, and they're doing excellent at the moment. It is two contrasting styles of play - I wasn't sure physically whether they were that dominant in our box or whether we were that soft, or a bit of both," Smith told reporters in his post-match press conference.
"I was disappointed with their third goal. For me it's a shoulder barge on Romaine [Sawyers] in front of their dugout, two minutes earlier Ryan Woods has got shoulder barged in their corner and nothing given. It just stinks of inconsistency. But then we've still got to go and defend it, and we didn't defend it well enough.
"They're very strong aerially in both boxes, and that is something we have to get better at. After the third goal every time the ball was thrown into our box there were panic stations at times, because they had that height advantage. But we're not competing against Thor. They're human beings and we've got to have a little bit more nous about us in our box.
"We were missing our most physical player in Henrik Dalsgaard, who we had at Millwall. Millwall are a really physical team and we nullified that on Saturday."
The defeat leaves Brentford 11th in the Championship table and now eight points adrift of the playoff places.