Valtteri Bottas won for the first time since April to reduce the championship deficit to Hamilton to 64 points with 104 available.
Pole-sitter Sebastian Vettel held off a late fightback by Hamilton to finish runner-up, just 0.4 seconds ahead of the Briton.
Hamilton will win his sixth world title in Mexico in a fortnight's time if he outscores Bottas by 14 points. If Hamilton wins, and Bottas is fifth or lower, the Briton will move to within one triumph of Michael Schumacher's record.
And despite Hamilton's frustration here, his Mercedes team were crowned constructors' champions for a sixth successive time.
London-born Thai Alex Albon finished a career-best fourth for Red Bull ahead of McLaren's Carlos Sainz.
Charles Leclerc was sixth after he collided with Red Bull driver Max Verstappen at the first corner. The Dutchman was forced to retire with the damage to his car.
Hamilton spent much of the 53-lap race at Suzuka questioning Mercedes' decision not to put him on the hardest tyre compound at his first stop.
Hamilton, who was running in third at the time, believed that would have provided him with the chance to get to the end of the race on a one-stop strategy, and a potential victory.
"How have I lost that much time?" said Hamilton, after losing 12 seconds to Bottas with Mercedes pitting the Briton three laps later than his team-mate.
"So I'm out of the race now? If tyre degradation is so massive why didn't you give me hard tyres? I'm basically out of the race. We're a pit stop behind."
Hamilton led with 10 laps to go, but, after reporting his rubber was wearing, he was brought in by Mercedes for an inevitable second stop.
That put him in third, four seconds behind Vettel. The five-time world champion hustled Vettel to the line, but he was unable to make a move stick.
"Congratulations to the team," said a clearly frustrated Hamilton. "It is well deserved. That is the main point."
The race started in a flurry of action when pole-sitter Vettel failed to get away from his starting blocks.
That provided Bottas with the chance to leapfrog both Vettel and Leclerc after the Ferrari cars had earlier locked out the front row following a qualifying session postponed from Saturday.
A fast-starting Verstappen sailed round the outside of Leclerc but was taken out of the race when the Ferrari driver bumped into him. The incident was due to be investigated after the end of the race.
Leclerc continued with a damaged front wing but after pieces of his car flew into the following Hamilton, the Monegasque stopped for repairs, putting him out of contention.
Vettel, who escaped action following an investigation into a false start, was the first of the leaders to stop on lap 17. Bottas followed in a lap later, with a two-stop strategy the order of the day.
Hamilton stayed out longer, but he, too, was unable to make his rubber last, ensuring Bottas would claim his third win of the year.