The 43-year-old, who is allegedly on the verge of joining the Swans, said that his top priority is his side's weekend match against BK Hacken.
"It's just speculation," Swedish broadcaster C More Entertainment quotes Potter as saying. "We have to focus on the game on Saturday.
"We've had a tough spring because of many reasons but we have the opportunity to finish the spring with 17 points if we win on Saturday. Let's focus on that."
Swansea dismissed their previous head coach Carlos Carvalhal at the end of the season after he failed to keep them in the Premier League.