Arsenal are allegedly considering the sale of Kieran Tierney as part of a summer overhaul by Mikel Arteta.
The left-back arrived at the Emirates last year in a £25m deal from Celtic, but Tierney has suffered an injury-plagued debut campaign in the Premier League.
Indeed, the 23-year-old has only turned out five times in the English top flight, and the emergence of Bukayo Saka at left-back could supposedly see the Gunners willing to part ways with Tierney.
Speaking on the Transfer Window podcast, Castles said: "There is a fit there [for Chilwell at Chelsea].
"I think it's important that Brendan Rodgers, when he was entering Leicester City and looking at last summer's transfer market, Chilwell was a player he was prepared to let go.
"He felt he could secure a substantial transfer fee for him and if he did the player he had in mind to purchase was Kieran Tierney.
"Obviously neither of those things happened, Tierney went to Arsenal instead and Tierney has struggled through injury, with the managerial changes at Arsenal as well, made just 11 appearances for them so far, just five in the Premier League.
"So when you're looking at a player who struggled in that first season like that, was bought for a substantial fee, he is at a club which has money problems, they need to make changes to their squad and they need to find revenue to do that.
"So in normal circumstances you would say no chance of getting a player from Arsenal who had been bought for a substantial fee and brought in as part of the last and most recent rebuild.
"Perhaps in these circumstances a deal would be viable for Leicester and I think it fits with Brendan Rodgers's assessment of the players and the way he has pragmatically looked at his squad at the past."
Tierney is under contract at Arsenal until the summer of 2024, but the Scotland international has only managed 11 appearances across all competitions for the Gunners.