Sunderland were on course to pick up a point at the Stadium of Light, until Wes Brown's own goal with 15 minutes remaining condemned them to defeat.
"It's harsh," Poyet told Sky Sports News following the game. "There have been four or five games since I came here that we couldn't win, that maybe we were very close to winning. That can change your season.
"It's the timings. Sometimes you feel the defeats are coming in a terrible moment. That's football. It's difficult to accept, but you have to try and the boys tried everything today. But that is the story of the season. Something always happens that seems to go against you - it could be an own goal, a referee's decision or a deflected shot.
"When you're at the top, it goes for you. When you're at the bottom, it doesn't. And it hurts. It hurts a lot. But that's football. It's difficult to accept, but you have to try and the boys tried everything today."
Sunderland face Manchester City on Wednesday night in the first of their two games in hand on the teams directly above them.