The Raiders have crumbled in the second half of their season as they've lost five out of their last six games to drop to 4-9 in their season.
Allen told reporters: "Listen, that's not what my job is. My job is to try to get this team better, get this team ready to play against the Kansas City Chiefs.
"I'm not going to talk about speculation or anything like that. I'm going to worry about winning a football game this week."
Allen has been the coach of the Raiders since the start of the 2012 season.