Hoyer briefly won the starting role with the team last season before suffering a torn anterior cruciate ligament, while Manziel was selected in the first round of this year's Draft.
The head coach told reporters: "I don't think it's insurmountable. I think that Brian is securely ahead of him right now, but we will compete and we will decide.
"The rookies haven't been here very long. They're playing catch-up from a playbook standpoint. At this point, we really weren't keeping score. We'll be much more apt to do that when we get to training camp.
"If a guy has a chance to be a starter, I would think you'd want to expose him to a starting defense."
The Browns have not reached the playoffs since 2002.