Whittaker saw red for throwing the ball at a Malmo player in retaliation to a mistimed tackle, while Bougherra was dismissed for an elbow while challenging for an aerial ball.
"The two sendings-off cost us dearly," said the Scot.
"The majority of our boys were fantastic tonight, I really could not have asked for any more than they gave me.
"I think indiscipline cost us the tie tonight," he added.
The reigning SPL champions will now play a qualifier to enter the Europa League.