Fans voiced their displeasure over the lack of investment being put into the club before Saturday's loss to Middlesbrough.
"I'm saying I'm unimpressed by the demonstrations of a few morons," said the 79-year-old.
"I ain't going anywhere soon. Some fans may not like me, or agree with me, but you're stuck with me."
The former Chelsea chairman became the controlling shareholder in the club in April with a 72% share.