Liam Brown put Gary Naysmith's men ahead with quarter of an hour played, but the hosts' Harvey Swann ensured it was all square 11 minutes later.
Blair Henderson had the final say of the half to put Edinburgh back in front and added another on the hour mark before he was forced off with an arm injury.
The visitors still held on for all three points to make it four wins since ex-Hearts and Everton ace Naysmith was appointed on March 9.
Stirling are level on points with Edinburgh but a place behind them in third after a hard-fought 1-0 victory at home to Stenhousemuir.
Jack Leitch grabbed the only goal four minutes after the break to move Albion onto 25 points while the visitors had Tommy Muir sent off late on.
Elgin looked like they would miss the chance to move into the top four until Darryl McHardy struck an 83rd-minute winner at Brechin.
Connor Barron put the basement club on top, but Kane Hester equalised two minutes into the second half before McHardy popped up to make it 2-1, helping Elgin leapfrog Stranraer.