The Ivory Coast international has had to wait seven months for his full debut but he made his mark as Celtic quickly regained top spot on goal difference following Rangers' 1-0 win over St Mirren earlier in the day.
There was some doubt about whether Bayo got the final touch on both occasions but his determination to get on the end of crosses from James Forrest and Kristoffer Ajer merited the recognition.
Callum McGregor fired a trademark long-range strike in between as Celtic made it 15 goals from their opening three league matches.
Bayo was handed the lone striker's role after working his way up to fitness following hamstring problems which had restricted the 22-year-old to three late appearances off the bench since arriving from Slovakian side Dunajska Streda in January.
The Ivory Coast international showed the Celtic fans he was worth the wait with a lively display and he was centimetres away from a hat-trick when he headed Forrest's cross off the inside of the post just before going off with cramp in the 78th minute.
Fraser Forster also came into the Celtic side after rejoining the club on loan from Southampton in midweek and the goalkeeper saved Conor Washington's 81st-minute penalty only for the Northern Ireland international to net the rebound.
The goal was a mere consolation for Hearts, who denied Celtic any chances in the opening quarter despite losing Michael Smith to injury early on.
Bayo got an opportunity after peeling off his marker to meet Ryan Christie's corner. Joel Pereira blocked the header at the expense of a corner.
Hearts' on-loan Manchester United goalkeeper did not look so assured when Celtic took the lead five minutes later.
Christie opened up play with a crossfield ball and Forrest delivered a first-time cross from the left wing.
Craig Halkett appeared to leave it for his goalkeeper but Pereira was slow in coming out and Bayo threw himself at the ball which went in off either the frontman or Christophe Berra.
Forster, making only his second appearance since Boxing Day 2017, was booked for hacking down Jake Mulraney outside his box after initially dealing with a dangerous ball forward. He made a smart save from Aaron Hickey's well-struck half-volley moments later.
Hearts threatened to cut their way through Celtic with the help of some sharp passing from debutant Glenn Whelan but Washington was flagged offside as Scott Brown slid in to block his shot.
Celtic soon went two ahead in the 54th minute. Forrest's run and Christie's instant pass gave McGregor time to shoot from 22 yards and he fired into the corner of the net.
Bayo was celebrating again six minutes later after cutting across Halkett to meet Ajer's low cross. The ball swept into the far corner after both men slid in.
The striker was greeted with a hug from manager Neil Lennon after making way for Leigh Griffiths, who had a frustrating afternoon as Hearts pushed forward for most of the closing stages.
They were lifted by Washington's goal after Brown was ruled by Willie Collum to have fouled Sean Clare as both men competed for a loose ball, but Craig Levein's side could not trouble Forster again.
Celtic now travel to Sweden for the second leg of their Europa League play-off against AIK before facing Rangers next Sunday.