Conor Hourihane's second-half header from Alan Hutton's cross was the only goal of the game as
Villa took all three points to push them further towards the Championship's play-off places.
Swansea were the better side for long periods, but they huffed and puffed and lacked an end product.
Swansea winger Daniel James went down in the box under a John McGinn challenge inside the first 10 minutes, but referee Roger East waved away home penalty appeals.
Bersant Celina then fired wide and Mike van der Hoorn's header was saved by Orjan Nyland as the hosts made a bright start.
Villa hadn't got started, but they then responded with the game's best chance. Against his former side, Tammy Abraham headed Anwar El Ghazi's cross down to McGinn and he fired on target, but the ball hit Swansea goalkeeper Erwin Mulder and stayed out.
Celina fired wide for a second time before Van der Hoorn led a Swansea breakaway. The centre-half intercepted an Abraham pass, ran clear, and then unleashed the flying Daniel James.
His cross found Oli McBurnie, but it was delivered a fraction late and the striker was crowded out by Villa's desperate covering defence.
Abraham, Yannick Bolasie and McGinn all headed wide at the other end as the two teams went to the break level with Villa finally having built up a head of steam.
Swansea captain Leroy Fer tested Nyland at the start of the second half and Bolasie's deflected effort saw Mulder forced to flap the ball clear.
Mulder was soon back in action to beat away two Hourihane free-kicks in quick succession as the game opened up.
Swansea swapped James for another wideman in Jefferson Montero in search of a response and it almost immediately coincided with them falling behind.
Villa full-back Hutton found space down the left and he stood up a cross which Hourihane headed home. Mulder got a hand to it, but the effort had too much power as it went in off the bar.
Swansea introduced Bony and Nathan Dyer as they looked for an equaliser.
Bony did work Nyland from range and McGinn fired in another shot, while Van der Hoorn's header from a corner rebounded off a post.
Roberts appealed for handball against El Ghazi, but Swansea were then given the perfect opportunity to rescue a draw. Competing for a high ball, Dyer was needlessly bundled over by El Ghazi and East rightly pointed to the spot.
Bony stepped up, but Nyland was more than equal to the task as he sent Swansea to a second straight defeat.