The 19-year-old defender becomes the club's first signing since their relegation from the Premier League, which was confirmed by a 3-0 defeat to Chelsea earlier this month.
Mahmutovic will officially join the club when the transfer window reopens on July 1 and will initially link up with Boro's Under-23 side.
"We're delighted to welcome Enes to Boro. He's a young defender with a good amount of experience for a boy of his age. He's a very quick, strong and aggressive defender. Technically he handles the ball well and has good distribution," academy manager Craig Liddle told the club's official website.
"Initially we see him coming into the Under-23 group and settling into the club while developing himself and the young players around him."
Mahmutovic is a two-time Luxembourg international and was part of the team that lost 3-1 to Netherlands in a World Cup qualifier last November.