The Rossoneri were keen to land the 30-year-old Brazilian, but have now backed down on the move after consulting a tax accountant.
"The tax problem? It's all true," Galliani told Milan Channel. "It's impossible.
"We asked the opinion of a tax accountant, who said it was 'too risky'."
Kaka joined Madrid from Milan in 2009.