St Mirren would have come off the bottom of the table by avoiding defeat, but Miller's late goal consigned them to another loss.
St Mirren started the match strongly, and veteran striker Steven Thompson had two shots blocked in the opening 10 minutes.
Former Killie winger James Dayton then saw his shot deflect off the crossbar in the 17th minute.
Eccleston again threatened just before half time, but his shot was saved by Mark Ridgers.
Dayton proved dangerous once again in the second half, but Samson was equal to his shot directed towards the top corner.
Josh Magennis then shot wide for Killie a minute later, and the former Aberdeen striker missed again with his head 10 minutes later.
Sean Kelly had a great chance to score the winner for St Mirren, but he shot over the bar from close range.
Substitute Miller grabbed the winner in the 87th minute, finishing from close range after Kilmarnock dissected the St Mirren defence.
St Mirren remain at the foot of the Scottish Premiership table, while Kilmarnock stay in eighth.