The 26-year-old claimed his first of the encounter with 10 minutes on the clock after Steve Arnold failed to collect a cross from the right. Bakayogo reacted quickest and managed to divert the loose ball into an unguarded net from close range.
Tranmere's advantage only last two minutes, though, thanks to Stevenage midfielder Luke Freeman, who curled a free kick from 25 yards out over the wall and inside the post in the 12th minute.
However, with what was their first meaningful attack after the equaliser, Tranmere regained the lead with 16 minutes played. The ball dropped to Ben Gibson inside the area and the Middlesbrough loanee drilled in the first senior goal of his career.
Rovers continued to dominate proceedings and with seven minutes of the half left to play, Bakayogo claimed his second goal of the clash. He held off a challenge from Mark Roberts, before going on to beat Arnold from 12 yards out.
Both sides had chances after the restart, but neither were able to convert those as Tranmere held out for all three points.