The 34-year-old returned to the Bescot Stadium in September 2013 following a stint in Cyprus with Nea Salamis Famagusta.
Since then, he has made 12 appearances for the Saddlers, but the winger is now on the look out for a new team.
"I am disappointed to have to let him go because I like him as a footballer and as a person," manager Dean Smith told the club's official website. "Unfortunately he was unable to break into the starting lineup and make the impact that he wanted to.
"We certainly wish him well and we will be doing all that we can to assist him in finding another club. On behalf of everyone at Walsall FC, I would like to thank him for his efforts during his two spells with the club."
Gray has played for a host of clubs, including Arsenal, Birmingham City and Coventry City.