The 25-year-old - part of Town's promotion-winning side last season - has penned a four-and-a-half-year deal to join Aitor Karanka at the City Ground.
Lolley had been linked with a string of Championship sides - including Sheffield United, Barnsley and Sunderland - after only managing to pick up six league appearances following the club's promotion to the top flight.
"We received an attractive offer for Joe; one that makes total sense financially for Huddersfield Town," Terriers coach David Wagner told the club's website. "Joe is also now at the point in his career where he wants to and needs to play regularly and that's something we cannot offer him at this moment in time.
"Joe has worked hard in every single training session and is a fantastic character. Everyone here wishes him all the best for the future."
Lolley joins a Forest side that are currently 15th in the Championship table, nine points above the dropzone and 13 points off the playoffs.