The big-serving American looked to be heading for a routine victory when he claimed the opening set 6-3, before going on to win a second-set tie-break 7-5.
Belgium's Malisse was visibly annoyed with some of the calls that were made by the umpire and his line judges and having gone two sets down, he vented his anger towards the man in the chair.
However, rather than being hampered by his frustration, Malisse broke Isner's serve for the first time early in the third set. Despite being broken back by Isner when he was on the cusp of winning the set, the 32-year-old responded with a second break to emerge a 7-5 victor.
There was very little so separate the two players during the fourth set, which ended up heading into a tie-break. Incidentally, it was the 50th breaker that Isner had contested this year.
Isner claimed the first mini-break, only to double fault with the score 5-4 in his favour. That allowed Malisse to take the advantage and he won the next two points, which left him on set point.
Isner rescued the situation to level up the score at 6-6, before both players went on a run of failing to take set points.
The outcome was finally decided as Malisse volleyed into the net at nine points apiece, with Isner then winning the next point on serve to advance through to round two, where he will take on Finland's Jarko Nieminen.