The Gunners are rumoured to be preparing a joint £35m bid for the Swans skipper and top scorer Michu.
"It doesn't get in my mind as I'm used to all the speculation now," Williams told the Daily Star.
"It's gone on for so long. It's there every day so I don't really pay too much attention to it. I just come in and do my job for Swansea.
"Everybody enjoys playing for this team. You all get touches of the ball because that's how we like to play. Certainly, nobody is in a rush to leave this club."
Swansea travel to Tottenham Hotspur this afternoon.