The Aviva Premiership leaders lost 24-12, with Jonny Wilkinson kicking all of the points for the French club.
"A 12-point winning margin seems as though it was comfortable for Toulon - but it wasn't," BBC Sport quotes the Saracens's director of rugby.
"There were a couple of times when we were three points down when we had some decent chances in their 22 and failed to capitalise. And Jonny Wilkinson kept on kicking goals from long distance. We did a lot of things well but not enough."
Toulon will face Clermont Auvergne in an all-French final in Dublin next month.