Their first opportunity fell to centre-back Emilson Cribari, who found himself unmarked inside the Berwick area with ten minutes played. The Brazilian attempted a back-heeled effort, which just went past the post.
Midfielder Ian Black also went close in the 38th minute with a volleyed effort that brushed the outside of the post.
However, just as it looked as though the two sides were heading into the break level, Little stole in with two minutes of stoppage time gone to give the away side the lead.
McCoist's side have picked up four points from their opening two matches of the new season.