© PA Photos
The home side were awarded a corner in the opening five minutes, which Rangers failed to deal with, and the ball deflected into the net off Anestis Argyriou.
Berwick's lead was short-lived, as the visitors won a penalty in the ninth minute after Dean Shiels was fouled in the area, before taking the spot kick to level the scores.
Rangers took the lead just before half time, with Wallace crossing the ball to Little, who volleyed it into the net from close range.
The win extends Rangers' lead at the top of the table to 25 points.