The 27-year-old believes that Switzerland were guilty of making too many mistakes at the back as France edged closer to a spot in the last 16 of the World Cup with one game remaining in the group stage.
"It's tough to take," Djourou told reporters. "We were expecting a better result. Or at the very least, a better performance.
"It was hard. Conceding two goals so quickly was a big blow for us. At this level, it's hard to come back from that. We kept on fighting but today France made us pay for our mistakes.
"With their high pressure, they made it difficult for us, but I think when you look at the game and the goals that we conceded you can see that they were all our mistakes. We made it so easy for them."
Switzerland take on Honduras on Wednesday, while Ecuador, who are level with Ottmar Hitzfeld's side with three points, take on France.