The Hoops were looking to recover some continental pride in Latvia following last month's humiliating Champions League exit at the hands of Ferencvaros.
Odsonne Edouard was back on duty after sitting out the weekend win over Livingston but the Frenchman showed little enthusiasm for the task of spearheading Celtic's attack.
And it was the man who replaced him with eight minutes left, Elyounoussi, who grabbed the all-important last-minute goal to seal victory and set up a clash with FK Sarajevo or Buducnost Podgorica next week.
Les Bleus Under-21 ace Edouard was supposed to be back refreshed having missed the Livi clash after complaining of tiredness on the back of a hectic schedule for club and county.
But he hardly exerted himself during some pretty lethargic opening exchanges at the Skonto Stadium.
The Scottish champions are well used to teams parking the bus, but Riga took it one step further after positioning a pair of double-deckers behind one of the goals so a couple dozen home supporters could cheer on their side.
They were off their seats after 21 minutes as Celtic got themselves so out of shape that the Latvians raced clear on a three-on-one.
Nir Bitton rescued the Hoops with a vital block before Vladimirs Kamess could roll in a team-mate but Lennon could be forgiven for having worrying flashbacks to last month's Champions League catastrophe.
The Parkhead manager had yet more to fret over when James Forrest went down with a calf injury after 32 minutes and had to be replaced by Jeremy Frimpong.
But at least by that stage his side were starting to create. Olivier Ntcham had a 25-yard drive tipped over by Roberts Ozols while a better connection might have seen Bitton net after Shane Duffy climbed high to nod down Christie's up-and-under into the box.
Edouard, having ambled around for most of the first half, suddenly shrugged himself into action as he slid McGregor in behind the Riga defence but Ritvars Rugins covered up well to deny Frimpong a tap-in at the back post.
The half-time break offered Lennon the opportunity to remind his team they had little margin for error in this one-off tie.
Christie showed some initiative but couldn't follow up his driving run with a finish past Ozols at the near post.
Portuguese ref Fabio Verissimo was feeling lenient as he let Kamess off with a yellow after a shocking rake of the studs down Scott Brown's Achilles.
Albian Ajeti had made a clinical start to his Hoops career with a goal every 40 minutes so far, yet he someone failed to convert from a yard out just moments after replacing Ntcham.
Frimpong got to the byline and fired the ball across goal but the Swiss striker got the ball stuck between his feet and Ozols pounced to smother.
With Edouard hooked, Frimpong was increasingly looking like the man who would make the difference with his surging runs down the left. Armands Petersons cleared another of his wicked low crosses off the line.
But there was no-one there to get a block on Elyounoussi in the 90th minute as Frimpong swept another cross into the box for the Norwegian to angle home a precious winner.