It proved to be a disappointing afternoon all round for the visitors, who had two players sent off as well as their manager Neil Lennon.
Rangers started the encounter positively and went on to break the deadlock in the 11th minute. Sone Aluko collected possession inside the Celtic half, before weaving his way past a number of defenders to slot underneath advancing goalkeeper Fraser Forster.
The away side reacted with some attacking threat of their own, which saw Rangers goalkeeper Allan McGregor forced into diving saves to deny Georgios Samaras and Anthony Stokes. Former Arsenal striker Stokes also struck the bar with a bouncing effort.
Celtic ended the half with ten men when Korean defender Cha Du-Ri was handed his marching orders for hauling down Lee Wallace, who was through on goal.
Boss Lennon was handed his marching orders shortly after the restart for showing dissent towards the match official.
Victor Wanyama soon followed his manager for a challenge on Rangers midfielder Steven Whittaker just after the hour mark.
Rangers made their numerical advantage count in the 72nd minute through substitute Andre Little, who converted the rebound after Forster had denied Wallace. The goal was Little's first touch of the match, having just replaced Aluko.
Wallace made sure of the points 13 minutes from time when he guided Steven Davis's through-pass beyond the reach of Forster.
Celtic began to mount a comeback late in the game as Scott Brown dispatched a penalty past McGregor after Carlos Bocanegra has brought down Samaras inside the area. American defender Bocanegra became the third player to be shown a red card during the match for his challenge.
The Bhoys then reduced Rangers' lead to one goal in stoppage time when defender Thomas Rogne headed in at the back post.
However, Lennon's side were unable to find a third goal to level up proceedings.
The defeat was only Celtic's third of the season.