The 27-year-old's contract runs out at the end of the current season, and the Saints are yet to come to an agreement with Graham over a new deal.
New Orleans have admitted that they could use the franchise tag on the tight end, but Graham's camp believes that he should be designated as a wide receiver rather than a tight end, which earns him more money for the one-year deal.
"I want Jimmy back!" Brees told the Dan Patrick Show. "I want Jimmy Graham back! It doesn't really matter what I say. That has no bearing on the rules of how he's designated."
Graham had an excellent year for the Saints catching 86 passes for 1,215 yards along with 16 touchdowns.