Farmer was a frontrunner for the general manager's position with the Dolphins, but he felt that finishing the interview process did not feel right.
In an interview with The Plain Dealer, he said: "I think Cleveland is primed and I think we're in a position to redefine the history books. Change is on the horizon and I didn't want to miss what's going to happen in Cleveland.
"It still is a dream of mine to become a general manager and run my own program and put together my own team, but the Miami thing just wasn't the right time for me."
Farmer has been with the Browns a year, after a six-year stint as director of pro personnel with the Kansas City Chiefs.