Elche made a bright start to the match, and came close to opening the scoring when Victor Rodriguez burst into the box before curling the ball towards goal, narrowly missing the target.
Borja Viguera could have found an equaliser for the visitors before the break as he tried to smash the ball in on the rebound, but his finish failed to test the keeper.
Elche were able to double their lead in the final seconds before the interval, though, as Adrian Gonzalez played the ball in for Jonathas, leaving him to rifle the ball home on the turn.
Benat Etxebarria and Inaki Williams were both on the attack early in the second half as the visitors tried to find a way back into the tie, but Elche's defence ensured that their chances were limited.
Athletic continued to look the more likely in the second half though, and gave themselves a chance inside the final 10 minutes when Ager Aketxe won a one-on-one with the keeper to slot the ball home.
The visitors were back on level terms with three minutes to go as Mikel San Jose hit the ball first time from 25 yards out, picking out the bottom-right corner.
Bilbao's late resurgence left them pushing for a winner, which they found in a stoppage time when Aritz Aduriz headed a cross into the path of Williams, leaving the striker to power the ball home, completing the late turnaround.
The result leaves Athletic seventh in the table, while Elche sit 13th.