The Bluebirds suffered their first Championship defeat since New Year's Day on Friday night, going down 1-0 to the league leaders in South Wales.
With the gap between Cardiff in second and Fulham in third potentially being cut to just two points on Saturday afternoon, Bamba insists that it is not the time to start panicking.
"If we keep winning our games we will be fine - we cannot worry about anybody else," he told Sports Mole. "I'm not too worried. Nobody expected us to be here so we just have to enjoy ourselves. I'm confident that we will be OK."
The tension was clear around the Cardiff City Stadium as both Madine and Hoilett stepped up to take their spot kicks, with the former being denied by John Ruddy down low and the latter striking the crossbar.
Asked if nerves got the better of his teammates, Bamba added: "Not at all. You get into football to play games like this. We had a good crowd and we gave a great effort - the fans could see that. We do not look down, we look up.
"We are obviously disappointed as we expected to take one of the penalties. You cannot get too high when you win a game and too low when you lose one. Now we have to go again. We are coming into the final six games now and need to win as many as we can."
Cardiff return to action on Tuesday evening with a trip to fourth-placed Aston Villa, who are seven points further back.