The 18-year-old Brit, who defeated Kim Clijsters in her last match, looked confident as three breaks of serve saw her take the opening set 6-4 against the former French Open champion.
Both players dropped serve early in the second before Li had the chance to convert a break point at 3-3. However, Robson's serve came to the rescue to see her hold.
With the contest entering a tie-break, the Chinese came through 7-5 to take the match to a third set.
Robson was dealt a slice of misfortune at 2-2 in the third as a forehand that would have given her a break point was wrongly called out.
However, it mattered little as Li soon hit long to put the Olympic silver medallist 3-2 ahead.
Robson then showed great composure from the baseline to break for a second time before going on to close out the match 6-2 to become the first British woman in 16 years to reach the fourth round of a Grand Slam, where she will play defending champion Samantha Stosur.