Kerr ended a four-game run without a goal for Saints with a header at the end of a tight first half before Kane headed in after the break.
Michael Tidser's effort for the Lowland League champions gave Kelty hope of claiming a draw but Saints held on.
St Johnstone manager Callum Davidson made six changes for the trip to New Central Park following his side's late defeat to Celtic.
A clash with the Lowland League side would have looked appealing on paper for Saints following a run of four straight league losses.
However, Kelty, managed by former Rangers and Scotland captain Barry Ferguson, were not short of SPFL experience in their side, former Falkirk midfielder Tidser and one-time Kilmarnock winger Kallum Higginbotham among those in their starting XI.
The top-flight side struggled to assert their dominance against a Kelty team who made sure their opponents knew they were in a game.
Callum Hendry should have done better with a header from a Craig Conway corner before Shaun Rooney had a shot saved from outside the box.
But the visitors opened the scoring in first-half injury-time when Kerr headed in Conway's corner.
It was a bitter pill to swallow for Kelty, who claimed that the set-piece should not have been awarded in the first place.
Liam Craig's half-volley appeared to clip the bar for Saints before they doubled their lead on the hour mark when Kane headed in Hendry's knockdown from close range.
Despite being largely outplayed, Tidser scored for the hosts with a 20-yard drive that swerved away from goalkeeper Elliot Parish in the 72nd minute.
However, the visitors held on to claim a much-needed win.