The game seemed to be drifting towards a goalless draw before substitute Mario Bilate arrowed a 25-yard strike into the top corner, much to the delight of the Tannadice Park faithful.
St Mirren followed up their opening-day loss with a 2-0 defeat at home to Hamilton Academical. Ali Crawford put the visitors ahead after 19 minutes and the striker had a big part to play in the second goal, as his shot deflected in off the unfortunate Marc McAusland shortly after the break.
Stuart Bannigan put them ahead five minutes before the break, and it was 2-0 in the 55th minute when Ben Frampah turned the ball into his own net less than 10 minutes after coming on for the visitors. Gary Fraser then netted a third before full-back Stephen O'Donnell rounded off a comprehensive victory in stoppage time.
Inverness Caledonian Thistle remain unbeaten following their 0-0 draw at home with Dundee.
Results in full: Dundee United 1-0 Motherwell, Inverness Caledonian Thistle 0-0 Dundee, Kilmarnock 0-2 Aberdeen, Partick Thistle 4-0 Ross County, St Johnstone 0-3 Celtic, St Mirren 0-2 Hamilton Academical