The 28-year-old is entering the final year of his deal with the Browns after making a positive impact in his limited starting time last season before an ACL injury ruled him out for the year.
Matters have now been complicated between the two parties as the club selected Johnny Manziel in the first round of the 2014 Draft, and he will compete with Hoyer for the starting quarterback position.
"It will be a very difficult deal to do. We're always open to talking with the Browns, but we're content to wait and see what happens down the road," Hoyer's agent Joe Linta told NFL Network.
Hoyer threw for 615 yards along with five touchdowns to three interceptions in his three games for the Browns in the 2013 season.