Tomljanovic secured the first blow by earning a break of serve in Radwanska's second service game of the opening set.
The Croatian continued to upset the third seed by securing another break point to take a 4-1 lead on the clay court.
Radwanska, whose chances of winning the Grand Slam were boosted when Serena Williams was knocked out early, battled to her first break point later in the set, but it wasn't enough as Tomljanovic took the lead.
The Croatian kept the momentum in her favour by breaking her opponent's serve instantly in the second set.
From then on, games went by way of serve, but it was a tough ask for Tomljanovic as she fought to save three break points to give herself a 5-3 cushion in the second set.
Tomljanovic's early break was enough to seal her a place in the fourth round at Roland Garros.