The visitors, who finished fourth in the Bundesliga last season, were dominant in the first half and took the lead after 16 minutes through Igor de Camargo.
After the break Norwich started well and were unlucky not to equalise when Jonny Howson's close-range effort was blocked off the line.
Chris Houghton brought on Simeon Jackson and Leon Barnett with five minutes left as Norwich chased an equaliser, but were undone on the counter attack.
Branimir Hrgota raced clear of the Norwich defence and rounded John Ruddy before firing into the empty goal with the last kick of the game.
Norwich begin their Premier League campaign next Saturday with a trip to Fulham.