The Baggies recorded their second victory of the season by beating the Premier League champions thanks to goals from Morgan Amalfitano and Saido Berahino.
Olsson told the club's official website: "It was a great game for us from the first minute. I'm very proud. It's a tough place to come, probably the toughest away game in the league.
"We played well. In the first half there was a little more focus on defending. In the second half we went for it a little bit and we're not going to apologise for taking three points.
"We knew they would be eager to attack in the second half and that opened up some space for us. People are always going to be a little surprised if we win at Old Trafford but, if you believe in yourselves, anything is possible."
The Baggies are 10th in the Premier League following the win.