Karim Benzema's first-half goal proved to be the difference for the European champions, although they produced another laboured performance against a team sitting 19th in the table and needed a double-save from Thibaut Courtois in injury time to clinch the points.
It nonetheless saw Madrid return to winning ways following their chastening midweek defeat to CSKA Moscow as they climbed up to third in the table and closed the gap on Barcelona - at least until the champions face Levante on Sunday.
Rayo shipped 10 goals on their most recent visit to the Bernabeu three years ago, and they may have been fearing a similar fate when Benzema opened the scoring after only 13 minutes.
The home side's early dominance paid off when Lucas Vazquez slipped a clever ball through for the French striker, who swept his finish into the bottom corner with aplomb for his sixth league goal of the season.
Such a rout never materialised, though, and Rayo responded well to going behind as Courtois had to prevent an own goal from Marcos Llorente before Emiliano Velazquez glanced a header narrowly wide when he should have scored.
The match was developing into another open and attacking contest between two sides who had shared 17 goals across their two meetings during Rayo's last season in La Liga, and Madrid were soon back on the attack when Marco Asensio was released clean through on goal only to be denied by Stole Dimitrievski.
Madrid looked dangerous every time they came forward during the midway point of the half, and Vazquez was the next to come close when his volley from a Toni Kroos cross was cleared just in front of the line by Velazquez.
The Rayo defender was having an impact at both ends of the field and again came close from a set piece moments later, but this time his header lacked the power to truly trouble Courtois.
The visitors' openness at the back forced them to curb come of their attack enthusiasm as the half wore on, although they needed the woodwork to come to their rescue with nine minutes remaining of the first half when Kroos's piledriver crashed against the post.
Santiago Solari may have wondered how his side were only one goal up at the interval, and another clear chance went begging six minutes into the second half when Dimitrievski thwarted Asensio's powerful half-volley from point-blank range with a brave stop.
Madrid were dominating possession, but they struggled to translate that into goalscoring opportunities for much of the second half and a correctly-disallowed goal for Sergio Ramos was as good as it got for the remainder of the match as far as the hosts were concerned.
Rayo were not creating much going forward themselves, but substitute Bebe did sting the palms of Courtois with a long-range strike which was too hot for the Madrid keeper to catch.
Courtois would prove to be Madrid's hero as Rayo nearly stole a point in injury time, with the keeper denying Alex Alegria's acrobatic effort before recovering to turn Velaquez's follow-up strike against Dani Carvajal on the line.
The visitors were left to rue a golden chance to end what is now a 17-match losing streak against their local rivals, but they can still be proud of only falling to a narrow defeat at the hands of a team that had scored 20 goals in their previous three home games against them.
Madrid, by contrast, were once again whistled off the field despite the win as they continue to struggle to find their best form ahead of their trip to Abu Dhabi to take part in the Club World Cup next week.
REAL MADRID (4-3-3): Courtois; Carvajal, Varane, Ramos, Marcelo; Modric (Valverde 92'), Llorente, Kroos; Vazquez, Benzema (Vinicius 78'), Asensio (Ceballos 71')
RAYO VALLECANO (4-5-1): Dimitrievski; Tito, Velazquez, Galvez (Ba 54'), Moreno; Advincula, Imbula, Medran (Bebe 54'), Santi, Garcia; Embarba (Alegria 73')