The 15-time major champion fought back from two sets down before Muller took a fifth match point to win 6-3 6-4 3-6 4-6 15-13 in four hours and 47 minutes.
It ended the Spaniard’s bid to win back-to-back French Open and Wimbledon titles for a joint-record third time.
Muller, 34, will play Croatia’s seventh seed Marin Cilic in the last eight.
“That was tough,” said Muller, who has reached his first Grand Slam quarter-final since the 2008 US Open.
“In the last two match points I just said give it 100%.”
Even world number two Nadal’s trademark competitiveness could not see off Muller in a gripping final set which took two and a quarter hours – more than half an hour longer than Roger Federer’s win over Bulgarian 13th seed Grigor Dimitrov.
Novak Djokovic’s match against unseeded Frenchman Adrian Mannarino, scheduled as the third match on Court One, was postponed until Tuesday as a result.