Midway through the half, Sergio Garcia was bundled over clumsily by Ze Castro, but the referee waved away the protests of the home support.
Just five minutes after the break, Larrivey turned the game on its head when he attacked a deep cross on the edge of the box and nodded it beyond the reach of Casilla, who had strayed too far off his line.
Casilla redeemed himself on the hour mark when he tipped Ruben Rochina's left-footed strike behind for a corner as Rayo pushed for another goal.
A third Rayo goal was disallowed after Bueno was adjudged to have been offside when tapping home from Larrivey's cross.
The miss proved costly, as Colotto rose highest from a corner to nod the ball into the back of the net and earn a draw for his side.