The Aussies currently have Alex Doolan at three, but Smith's recent half century against Pakistan in Dubai has called on some to suggest that the 25-year-old should move up from fifth.
Smith impressed by scoring a century and 77 while at three in the one-day side against Pakistan, but Clarke has insisted that the Sydney-born cricketer shouldn't be rushed in the Test game.
"It's a fantasy right now," cricket365.com quotes Clarke as saying. "In the future I think Smithy can definitely go higher up the order.
"He's just just starting his Test career though. He's playing really well but we've also got to give him plenty of time."
Australia were beaten by 221 runs against Pakistan in the first Test in Dubai last week.