Tareiq Holmes-Dennis was a constant attacking threat from midfield and saw two efforts blocked, but Dunn was lively on the counter for Cheltenham and Ryan Clarke had to be alert to smother the ball at the feet of the Liverpool loanee in the 27th minute.
Burns came close to adding a second 10 minutes before the interval, but his fizzing strike flew inches wide of the far post.
Four minutes before the break, Dunn's lively first-half performance was rewarded with a goal as he guided Lee Vaughan's pass into the bottom corner in the 41st minute.
Two minutes into first-half stoppage time, fellow Liverpool youngster Stewart netted for the visiting side to ensure they entered the half-time break a goal ahead.
Holmes-Dennis saw a strike deflected narrowly over the bar early in the second half as Oxford looked for a way back into the game, but Dunn almost put the visitors two goals ahead when he rounded Clarke, only for his goalbound effort to be hacked away.
Carson then denied Hylton from close range before John Campbell's flick was cleared off the line by Vaughan in the closing stages as Cheltenham held on to secure three vital points.