Craig Gunn scored from the spot to put the hosts under pressure.
However, Albion eventually got themselves back into the game after the restart, when Mark Lamont made a menacing run into the box to fire a low shot into the far corner.
Lamont continued to terrorise the City defence, as it was his cross that set up Doris's goal to give Stirling a 2-1 lead with 10 minutes remaining.
Elgin failed to contain Stirling's wingers for a second time shortly after, as Smith latched on to the end of Sandy Cunningham's cross to complete a late comeback.
The win puts Stirling in sixth, while Elgin keep their place in third.