Brendan Rodgers's side stormed past Linfield in the previous round to raise hopes of reaching the group stages, but their lack of striking options proved damaging as Rosenborg took a goalless draw back to Norway for the second leg.
It was Celtic who enjoyed the bulk of the possession during the opening stages, but some resolute defending prevented them from creating any clear chances on goal, with a header from Kristoffer Ajer the best they could muster.
Indeed, it was Rosenborg who came the closest to opening the scoring in the first half when Craig Gordon was forced into a diving save to keep out a free kick from a dangerous position.
The absence of injured Moussa Dembele and suspended Leigh Griffiths was proving to be problematic for last season's domestic treble winners, and they went into the break having failed to test the opposition goalkeeper aside from a comfortable Kieran Tierney header.
A Rosenborg free kick again almost provided the opener nine minutes into the second half, but again Gordon was there to turn the ball behind.
James Forrest then sent an effort over the crossbar from the edge of the box before former Arsenal striker Nicklas Bendtner had his first sight of goal, nodding a header off target with a little over half an hour remaining.
The visitors should have taken the lead with a valuable away goal moments later, but Yann-Erik de Lanlay failed to convert from close range before Bendtner drew another stop from Gordon with a curling strike.
Celtic's first clear opening of the second half arrived with just under 20 minutes left when Forrest again found space to shoot, only to this time drag his finish narrowly wide of the post.
Ultimately there was no way through for the hosts, who must now win in Norway to progress into the playoff round.