Mayer, 29, is currently the world number 25 but he was unable to beat a player ranked 26 places below him.
Matosevic claimed two breaks of Mayer's serve during the opener which allowed him to open up a one-set lead with a 6-2 success.
He broke again early in the second set to put himself in the ascendancy. A second break soon followed and despite Mayer claiming one of those back, he was unable to prevent Matosevic from serving out 6-3.
He will now take on Kevin Anderson for a place in round three.