Pogba's recent performances for the Red Devils have been called into question, and the Frenchman started Wednesday's Champions League last-16 first leg against Sevilla on the bench.
Ander Herrera's injury is expected to see Pogba return to the starting XI against Chelsea, but Conte has said that "it will be better" for the Blues if the former Juventus midfielder is not selected.
"We are talking about a top player, about a fantastic player," Conte told reporters. "My expectation is to see him on the pitch, if he stays out [of the XI] it will be better.
"As I said before, we are talking about a top player. Every coach has to find the right way to manage him. But I think that Pogba is showing at United that he is a top player."
Chelsea will enter Sunday's match two positions and three points behind second-placed United in the Premier League table.