The 28-year-old started three games for the AFC North side in 2013, but missed the majority of the season after tearing his ACL.
The Browns' new head coach told reporters: "I'd always admired Brian from afar. I think he's proven he can be successful. But at the same time you are always looking to make your team better.
"[General manager Ray Farmer] talked about this the other day. We evaluate players based on what they've done in the league.
"We have our opinion. That's something moving forward as that position shakes out, will work itself out.
Hoyer is expected to face competition from at least one rookie quarterback for the starter's job in 2014.