Goals just before and just after half time gave the Hornets a surprise 2-1 lead over the Premier League champions, only for a Michy Batshuayi brace and a Cesar Azpilicueta header in the space of the final 20 minutes to hand the hosts all three points.
The Hornets could have moved up to third in the table had they carved out a win and speaking after the final whistle, Silva admitted to being "frustrated" that his charges did not kill off the game sooner.
"It is a really unfair result. It didn't reflect the match and what we saw on the pitch," he told Sky Sports News. "Our team came here and controlled most of the match. We didn't create many chances in the first half, it is true, but we controlled the game and Chelsea started to play on the counter-attack.
"We played our game and in the second half for 40 minutes there was one team playing on the pitch. After 2-1 we had clear chances to finish the match.
"I'm proud of how we played, but of course I'm really frustrated. It showed our way and we need to continue in this way."
Next up for Watford is a home encounter with Stoke City next weekend.