Quins, who won their opening match of the season against London Irish, have not defeated Sarries since 2012.
With such a disappointing record against their rivals in recent times, Brown admitted that his side need to be at the top of their game if they want to end that run.
"Our level has to stay constant from the first minute to the 80th, even when teams start trying to slow our ball down or refereeing decisions don't go our way," he is quoted as saying by the Rugby Football Union's official website.
"We want to be top four and the only way to do that is to play the whole game. We'll have to do that against Saracens because they'll sting us if we don't.
"They are renowned for their physicality, so it'll be about fighting fire with fire. We're on our home patch, so we need to be the enforcers straight from the off."
Harlequins welcome Saracens to Twickenham Stoop on Friday night.