Danny Cruz was close to putting Philadelphia ahead in spectacular fashion in the 11th minute but the midfielder's long-range effort was superbly saved by a diving Matt Lampson.
Both sides struggled to penetrate the other's defences but the hosts found the opening goal from a free kick after 29 minutes.
Carlos Valdez curled the ball over the wall from 30 yards to give his side the lead with a brilliant effort.
Columbus reacted well to the goal and the visitors were level before the break when Josh Williams headed past Zac MacMath from a corner with four minutes of the first half remaining.
Lampson was called into action in the second period to deny Amobi Okugo after the defender found space inside the penalty area.
Columbus were reduced to 10 men when Williams was shown a red card for violent conduct after starting an altercation off the ball, but the visitors shocked Philadelphia in the fifth minute of stoppage time as Gaven scored a dramatic late winner.