The visitors were quickest out of the starting blocks and had a penalty appeal turned down just three minutes in when Robbie Simpson was brought down in the Swindon box.
The referee pointed to the spot and showed the Robins goalkeeper a red card, but James's penalty was saved by substitute stopper Cameron Belford.
The visitors went on to double their lead just after the restart as Simpson's saved shot was bundled home by Chris Dagnall.
Nevertheless, the hosts rallied round and pulled a goal back just after the hour when Anton Rodgers's free kick found the back of the net.
Williams was a constant threat for the Robins and came close to bringing his side level on two occasions midway through the second half, but one effort was saved by Alex Cisak and the other flew inches wide.
Orient were then undone in the final five minutes of the game when Swindon were given a penalty, which Williams converted to level the scores.
Combined with Colchester United's 1-0 win at Preston North End, the result saw the Brisbane Road outfit end the season third from bottom. Swindon were assured of a playoff berth before the game.