It was a miserable game for Villa who are now without a home win in their last five games.
It is also the third successive season that Villa, seven times winners of the FA Cup, have failed to reach the fourth round.
The absence of Tammy Abraham, who cannot play in the competition under rules of his loan agreement from Chelsea, was a major handicap for Villa. Scott Hogan was drafted in as a replacement for his first start of the season and the Irishman should have made a dream start after only 18 seconds.
Anwar El Ghazi produced a low cross from the left to the near post where Hogan glided his shot wide from close range.
It proved to be an expensive miss as Swansea took the lead in the second minute. Baker-Richardson was on hand to latch onto a right wing cross from Mike Van Der Hoorn and squeezed home his close range shot past Villa's new goalkeeper, Lovre Kalinic.
Hogan had another opportunity to open his account midway through the half but Erwin Mulder blocked the effort which was subsequently cleared off the line.
Newcomer Kalinic did well to keep out a speculative long range effort from Joe Roden as Swansea threatened to increase their lead against Villa's insecure defence.
As the game wore on, Villa's fans became increasingly disgruntled with the performance of their team.
Swansea, on the other hand, were generally more positive with Dyer proving to be an inspiration.
Two minutes into the second half he demonstrated his confidence by increasing Swansea's lead when he moved onto a through ball from skipper Leroy Fer and scored with a low drive past Kalinic.
Baker-Richardson made way for Fulton in the 70th minute and eight minutes later the substitute added Swansea's third goal as Villa's defence failed three times to clear their lines.