Smalley slid the ball into the net on six minutes after collecting a clever reverse pass from Ryan Williams.
The hosts' goal came from their only shot of a quiet first half and the U's were fortunate to hang onto their lead after the break.
Torquay dominated the second period and came closest to equalising on 90 minutes when Kevin Nicholson's free kick was pushed onto the crossbar by goalkeeper Ryan Clarke before being headed off the line by Jake Wright.
Wright's heroics came at a price, though. He had to be stretched off the field after being injured in a collision with Shamir Goodwin.
Oxford climb to sixth in the League Two table, while Torquay remain 23rd.