Groves announced his withdrawal shortly after he was made mandatory challenger for Robert Stieglitz's WBO title.
The 23-year-old blamed a back injury, which he said that he picked up during sparring with Eamonn O'Kane.
However, Anderson has said that he believes Groves has cancelled as he is unwilling to risk defeat, which may scupper a potential title tilt.
"It's brutal because of all the effort we have put in," Anderson told Boxing Scene. "A lot of people sacrificed a lot of things. It's me that gets in there and fights, but everyone in my life makes sacrifices - I am gutted.
"I think [Groves's camp are] going to have to look in the mirror and know they've dodged me for no other reason than they don't want to take the risk. I don't believe in my heart that he's injured. I don't think they wanted to have this fight, it is too risky and they want the [Stieglitz] fight because it is a bigger fish.
"If so then that is absolutely shocking, it is a terrible thing to do given the build-up and how much respect we've shown. The only time I'll get to fight Groves now is when I take the world title from him."