Twelve players have left the Welsh side since they dropped out of the Premier League last season, while only four new faces have arrived as they begin life back in the Championship.
The transfer dealings have frustrated fans, who chanted 'where's the money gone' and 'we want Jenkins out' during their 1-0 win over Preston North End at the Liberty Stadium on Saturday.
"I understand frustration. I can't control too much of what's happened over the past, and you understand in that context there's disappointment. I think the club has spent money in the past and money in itself doesn't necessarily solve the problem," Potter told reporters.
"We have to solve it in a good way, in a sensible way. In a way with an idea, and we have to make sure that the principles – in that the money is spent – are solid.
"There's a chance we'll get some players, but again they have to be the right ones. This time last year there were two loan signings, big loan fees, big transfer fees and players. Everyone's excited, but they don't turn out as everyone would like. It's about the right people, the right profile of player, and then working with the group."
Swansea have won their opening two games in the Championship and travel to Birmingham City on Friday night.