The midfielder struck again just after half time before Jack Sampson completed the scoring in the 56th minute.
"We started the season with three wins and now we've had three defeats with the same set of players so it's quite puzzling," PA quotes Brown as saying.
"If we can now win the next three again, I'll be very happy but I'm very disappointed by this display.
"The players were below par and their decision making wasn't good enough. That includes senior players who we brought back in here, but we didn't deserve anything from the game and there still wasn't enough nous in our team."
Southend are currently seventh in League Two.