The 46-year-old, who acted as Scotland interim coach during this year's Six Nations, was previously in charge of Gloucester Rugby for four years until 2009.
"I honestly didn't think I would go back," Ryan told Sky Sports News. "I think over the years it was no secret I fell out of love with the game a little bit. I lost confidence in my final year at Gloucester.
"But time is a healer and the opportunity to work with Scotland lit a few fires and raised a few questions in my head that I hadn't considered.
"There is something burning inside. I like competing and that means putting yourself in the firing line and taking on the challenge."
Ryan will start his new role with Worcester at the end of the season.