The welsh side began well, with Tom Brown scoring a try within six minutes. Lewis Robling converted to give Newport a 7-0 lead.
Bayonne did not take long to hit back as two penalties in five minutes from Benjamin Boyet reduced the deficit to one.
Tom Prydie then scored three penalties of his own without reply to give the home team a 16-6 lead.
The visitors again responded though, this time with a try as Mark Chisholm went over. Boyet converted to ensure the two sides went in at half-time level.
Bayonne took the lead for the first time early in the second half when Boyet claimed another penalty to put them 16-19 ahead.
Prydie then scored his fifth penalty to bring the scores back level, but only briefly as Boyet then scored again. The Frenchman had time to miss another penalty before the end as the Welsh side lost out.