The 16th seed and the double tournament champion were out on Court 1 for just under five hours, making it the longest match of the Grand Slam so far this year.
Muller's two-set lead was pulled back by Nadal, but the Luxembourgish player came out on top 15-13 in the deciding set of the fourth-round clash.
"I didn't feel like I was playing bad in the third and fourth sets so told myself to keep it up and if it doesn't happen then he is just too good," Muller told the BBC. "I started serving better and in the end it was just a matter of a few points here and there.
"I haven't really realised what has just happened, it is a great feeling. I was worrying it would be over soon because of the light and was thinking what to do when come back the next day and finish this match. I'm just glad it's over."
Before today, Muller had never reached the quarter-finals at Wimbledon.