Carter took an early 2-0 lead, but the two-time champion of the competition bounced back to win four frames in a row.
The 33-year-old Englishman reduced the deficit by winning the following frame, but Higgins won the next two to go through to the next round.
Former world champion Dott won the first two frames, but Maguire came back strongly to level the score at 2-2.
In what proved to be a topsy-turvy contest, the 35-year-old then fell behind only to win the following two frames and take a 4-3 lead.
The contest eventually went to a decider that Dott won to clinch it 6-5.