Saiz, signed for a reported £3m from Huesca in July, marked his first appearance in England with a hat-trick, but Brown claims that the Spaniard spat at Vale defender Joe Davis shortly before half time.
"I think the boys lost focus a little bit," Brown, who spent three seasons at Leeds before departing in 2014, told Sky Sports News.
"The guy who scored three goals spits in Joe Davis's face in the 40th minute and it is very hard to take.
"So that certainly knocked us off balance. We asked their staff to take him off, they ignored that and he scored three goals. That is why it was getting a bit heated down in the technical area."