Speculation in Spain has claimed that Ivanovic is high on the list of Real targets as Jose Mourinho looks to strengthen his squad for next season.
Ivanovic, 28, who was a starter in Serbia's 2-0 defeat against Spain on Saturday, told Marca that to think about switching to La Liga would be "complicated".
"I know what has been written about my future and that Madrid are following me, but now is not the time to talk about this," he said.
"I am focused on the national team now and I am very happy at Chelsea.
"I won the Champions League just one week ago and to think about changing clubs now is complicated. I am very happy in London."
Ivanovic, who missed the Champions League final through suspension, joined Chelsea from Lokomotiv Moscow in 2008 for around £9m.