The Scottish Championship side arrived in Copenhagen trailing 1-0 after the first leg and, despite overturning that deficit, they ultimately fell short in the shootout as Brondby set up a meeting with Hertha Berlin in the next round.
Visiting manager Neil Lennon was forced to watch on from the stands having been handed a one-match touchline ban for his behaviour in the first leg last week, leaving academy manager Eddie May and assistants Garry Parker and Grant Murray to take charge in the dugout.
Hibs got off to a better start than they did a week ago when they found themselves behind after just 16 seconds, but Brondby did create the first chance of the match nine minutes in when Christian Norgaard sent his header over the crossbar.
Hibs did begin to grow into the game, but the main sights of goal continued to fall the way of the home side and Ross Laidlaw was once again relieved to watch a Lebogang Phiri effort curl over the crossbar shortly before the half-hour mark.
Laidlaw was finally called into action towards the closing stages of the half when he tipped Norgaard's long-range drive over the top, and Hibernian managed to hold on until half time with the score on the night still goalless.
The visitors knew that they needed a goal, though, and they had their first clear chance in the early exchanges of the second half when John McGinn's cross found David Gray at the back post, but he couldn't keep his header down.
Lennon's side were beginning to offer more going forward than they did in the opening 45 minutes, although Frederik Ronnow was rarely truly tested in the Brondby goal and comfortably collected McGinn's looping header 10 minutes after the restart.
Ronnow was beaten shortly after the hour mark, though, as Brondby failed to adequately clear their lines from a corner, allowing Darren McGregor to keep the ball alive for Gray to scramble it home.
The hosts looked for a response with the aggregate scores now level, but neither Phiri nor Teemu Pukki could hit the target with two chances in as many minutes.
Hibs did an admirable job of limiting Brondby's clear-cut chances, though, and the closest the hosts came to avoiding extra time was in the 90th minute when Larsson's shot flew wide of the target.
The match did go to an additional 30 minutes, but the first period of extra time passed by without any clear chances as both sides were forced to settle for long-range efforts.
McGinn tried to win the match in spectacular fashion with just five minutes remaining when he went for goal from 10 yards inside his own half, but his ambitious effort went off target and in the end the two sides were forced to settle for penalties.
Brondby gained an early advantage in the shootout when McGinn missed the visitors' first, and that proved to be decisive as the hosts were faultless from the spot, with Frederik Holst scoring the winning penalty to send his side through.body check tags ::