Erasmus adjudged Tim Bresnan out caught behind in the latter stages of the second day at Old Trafford, despite replays showing that Peter Siddle's delivery deflected off his trousers on its way through to Brad Haddin.
The South African official was also at the centre of attention during the first Test at Trent Bridge, where, acting as the third umpire, he contentiously overturned an unsuccessful lbw appeal against Jonathan Trott even though the DRS referral system seemed to suggest that the England batsman had hit the ball onto his pads.
There have been several other debatable calls made by Erasmus during the series, and Boycott believes that the ICC should replace him to avoid further controversy.
"What worries me is Marais Erasmus, the umpire," Boycott told BBC Sport.
"He's had a shocker this series, I'd honestly be more worried about him than by the Australian bowlers. If I were the ICC, I wouldn't defend the indefensible, I'd give him a rest."
Bresnan's dismissal helped Australia reduce England to 52-2, with the tourists leading by 475 runs at the close of Friday's play.