The Black Cats sit seven points adrift of safety at the bottom of the table following their 5-1 defeat at Tottenham Hotspur on Monday evening, but they do have games in hand on their relegation rivals.
"Their player has got behind me for the first and second goals," Brown told Chronicle Live. "I should be closer and then I just couldn't get there.
"We can talk all we want to about having to cut out the mistakes, but we have to do it. We have to do better when we're going forwards too. We're not helping ourselves.
"We've just got to keep going. The table doesn't lie, it's bad. But we've got to keep going. That's all you can do."
Sunderland host Everton on Saturday.