The midfielder, 31, has penned a two-year deal with the Championship club following his release from Leeds United.
Green spent the second half of last season on loan at Ipswich Town.
"We are delighted to have signed Paul after several chats with his representatives," Rotherham boss Steve Evans told the club's official website.
"The lad had a magnificent finish to the season with Ipswich Town so it wasn't a surprise to find there was significant interest in him, so we are very pleased to add him to our squad."
Rotherham get their Championship campaign underway at Derby County on August 9.