Holloway was manager of Palace last season and watched Williams break into the first team, helping the club achieve promotion to the Premier League.
The recent arrival of Thomas Ince at Palace has led to speculation that Williams will be allowed to leave on loan for the rest of the season, but Holloway isn't interested as he knows that there is little chance of Williams staying for any longer.
"Jonny is a good little player but I don't know where he would fit in here," said Holloway.
"I need to get long-term things for us and borrowing young players can be very dangerous.
"What I have never liked doing is improving other clubs' players only for them to go back at the end of their loan."
With 19 games of the season left, Millwall are just above the Championship drop zone in 21st place, two points clear of the bottom three.