The visitors started the game well and Josh Williams went close with an early effort after being teed up by Gláuber but Dan Kennedy was equal to it.
Williams was in the action again minutes later but this time he couldn't find the target as he dragged his effort wide, while Jairo Arrieta also saw a long-range shot saved by Kennedy.
Julio Morales has Chivas's best chance of the half but his shot was comfortably saved by Andy Gruenebaum.
After the break, Columbus took the lead when Arrieta played through Federico Higuain, who made no mistake in slotting the ball into the net.
Miller Bolanos had a chance to get Chivas level almost immediately but he couldn't beat Gruenebaum, who made a good save from close range.
The hosts kept pressing but it left them open at the back and Columbus took advantage of this late in the game as Williams finally got on the scoresheet, drilling an effort into the bottom corner.
Dominic Oduro sealed the three points as he slotted the ball home from close range following a blistering counter-attack by the visitors.