Hibernian came close to taking the lead before the break, when Jason Cummings saw a looping header tipped over by the keeper.
Scott Robertson nearly headed the hosts into the lead in the opening minutes of the second half, but put his diving header at the near post wide.
The home side took the lead on the hour mark, after James Collins latched on to a loose ball inside the box following a corner, and fired a strike into the roof of the net.
Brad McKay should have levelled for the visitors just five minutes later, but headed wide when he was left unmarked from a corner.
In the 71st minute Hearts equalised, as David Smith saw a powerful strike spilled by the keeper, before reacting quickly to fire the loose ball into the net, taking a deflection of Ryan McGivern on the way in.
The home side retook the lead in the 83rd minute, when they were awarded a penalty following Jordan McGhee's foul on Lewis Stevenson. Craig stepped up and slotted the ball into the net, sending the keeper the wrong way.
Kevin McHattie forced Williams into a save in stoppage time with his powerful strike, but Hearts were unable to find the net, as Hibernian held on to claim victory.