The midfielder was part of Boro's youth team, watching from the stands as the Italian netted after just 42 seconds in what turned out to be a 2-0 defeat.
Downing said that he is hungry for revenge against Di Matteo and Chelsea ahead of this weekend's FA Cup final.
"I've hated him ever since!" Downing told the Daily Mail. "We had got tickets off the club, who took us and our families down on the coach. It was a good day out but with bad memories of the final.
"What did I call him at the time? Nothing I could really repeat."
Liverpool will take on Chelsea in this season's FA Cup final on May 5.