Hart endured a difficult day at St Mary's as he allowed Rickie Lambert's shot to trickle through his legs for the second goal before being beaten by Gareth Barry when the midfielder put through his own net late on.
Mancini believes that Hart needs to be accountable for a series of below-par performances in recent weeks.
He told reporters: "Joe has made too many mistakes for us. He has not been good enough.
"It was a really bad mistake [for the second goal] from Joe. It was not good enough. It's not strange because he's done a few mistakes this year."
City have the welcome distraction of an FA Cup tie in their next match when they host Leeds United at the Etihad Stadium.