The Dragons had looked likely to take all three points in Cardiff following Gareth Bale's opener, but they were forced to settle for a point as Aleksandar Mitrovic headed in late on to deny Chris Coleman's charges.
"You want to take the three points, we fought well today but credit to Serbia. We are still undefeated, let's keep positive," said Ledley after the game, according to BBC Sport.
"Gareth Bale had just hit the post, if that goes in it's over. We are proud of our performance, it's Ireland away next and that's a must-win."
Wales currently sit third in Group D with six points, having played four qualifying fixtures so far.