Last season's beaten finalists were on top for long spells at Anfield but still needed a stoppage-time winner from substitute Roberto Firmino to overcome the Ligue 1 giants in Group C.
It looked as though PSG might have rescued a draw when Kylian Mbappe levelled things up eight minutes from time, but Milner was full of praise for Firmino after he ensured that all three points went to the hosts.
"It is the perfect start for us. Hopefully we can build on that," Milner told BT Sport.
"I think it would have been very disappointing not to win. They didn't create too many and we had a number of chances.
"I can't say enough good things about Bobby. To come off the bench after his injury at the weekend is incredible really."
Milner was also on the scoresheet with a penalty for Liverpool, meaning that his last 10 goals have now all come from the spot.