The Scot spent three years in charge of Norwich City before leaving for Villa Park last summer.
He said he enjoyed his time in East Anglia, but he is now looking to the future.
"I think if you get caught up in my connection there it deflects from the game," Lambert quotes Sky Sports News as saying.
"I had three fabulous years there and took a team from League One, built it up, the crowd bought into it, the lads I worked with were fantastic.
"It's another game and I'm looking forward to going down there. I have fantastic memories there but it has gone."
Lambert took charge of Villa in June 2012.