Amand Gnanduillet struck the winning goal six minutes into injury time and cancelled out Freddy Eastwood's equaliser.
Prosser has now revealed that Gnanduillet was his responsibility to mark.
"I've held my hands up, it was my man for the goal and I lost the flight of the ball when the corner came in," he told The Echo.
"We're all gutted and frustrated not to have got a point because that's the least we deserved."
The defeat was Southend's first in the league this season.