The former world number one was playing just his fourth match since undergoing hip surgery at the start of the year, while the 31-year-old's appearance was his first since opting to withdraw from Wimbledon.
Murray took time to grow into the match and comfortably lost the opening set, but the three-time Grand Slam champion's fighting qualities came to the fore during the remainder of the contest as he gradually wore down his American opponent.
The Scot took the second and gave himself a chance to serve for a place in the second round, but nerves - as acknowledged by Murray after the match - took over as he saw six match points come and go.
However, the seventh was converted when McDonald went long from the baseline, and it enabled Murray to celebrate his triumph after two hours and 37 minutes.
Murray now faces a last-16 showdown with British number one Kyle Edmund, with that encounter to take place on Wednesday.