Dale Hilson put Arbroath in front with a brilliant free-kick four minutes into the second half in Dingwall.
But Stewart headed home in the 65th minute and soon netted the winner from the penalty spot.
County suffered a blow when Callum Morris limped off injured and they continued to be frustrated.
Hylton lobbed wide and he and his fellow wide man Regan Charles-Cook could not quite find the end product to some promising build-up play.
Arbroath had enjoyed some decent moments in the first half but they stunned their hosts early in the second half when Hilson netted in style.
County levelled 20 minutes into the half when Stewart headed home at the back post from Randall's cross.
The home side were soon ahead after Gaston was adjudged to have brought down Charles-Cook after some sloppy defending kept a County attack alive.
Stewart netted to earn County their first win of the tournament.