Yan fought back from 3-1 and 5-3 down to claim a 6-5 win on his Masters debut and stun the Australian, who had swept to his third career UK Championship crown last month.
It was a remarkably mature performance from the 20-year-old Yan, and it hinged on a simple missed black by Robertson in the fifth frame when he was on the verge of going 4-1 in front.
Instead Yan proceeded to clear to pink to reduce the deficit to 3-2, and despite falling behind again he showed nerves of steel to drag the match back into a decider.
Yan, whose best major performance was a run to the 2019 UK semi-finals, established an early lead with a break of 35 then produced some superb safety play to defy his opponent and reach the last eight.
“I am quite surprised that on my debut Masters appearance, I have also beaten Neil Robertson, who has just won the UK Championship. I am very happy with the result and also playing so well in the decider,” Yan said, quoted on the World Snooker Tour website.
“I just go with the flow and do what I am supposed to do. That is what I did in the decider.
“Neil is the sort of player that takes a chance if you give it to him. I just played as much safety as I could.
“It was quite funny that all the colours were on the cushion. I did my best to mess up the table.
“I will do the best I can in the next round, but I have already gone beyond my expectations. I will try my best to relax and enjoy the match.”
Robertson was impressed with the 20-year-old’s display.
“Yan did really well, especially in that last frame,” the 2010 world champion said.
“It was like playing Steve Davis in his prime, tactically he was very good. He is fully capable of winning these events.
“When you compare him to the other Chinese players he doesn’t quite score as heavily as the others.
“If I was a bit clinical, I could have won quite comfortably. I didn’t take enough of my chances.”
Tuesday’s late match saw Stephen Maguire defeat three-time world champion Mark Selby 6-3.
Maguire swiftly moved into a 2-0 lead with breaks of 66 and 57 before Selby responded to level after two half-century clearances of his own.
A break of 99 helped Maguire regain the initiative, only for Selby, whose last of three Masters crowns came in 2013, to take the sixth frame.
Former UK Championship winner Maguire, though, built momentum again as he moved to within one of victory – which he secured with a clearance of 61 to book his place in the quarter-finals.