South African official Craig Joubert awarded 28 penalties during the match at Murrayfield, which is a record for an international match.
"We're very frustrated and I've said to the team we've got to acknowledge our ills in this and we have to look at ourselves and at what we are doing wrong," Johnson told reporters.
"We only got our first lineout in the second half and that's telling. We played once again with great heart and showed great resolve but we can't keep doing it.
"We've got to look at ourselves, I won't be blaming anyone else."
Scotland take on France in Paris in their next encounter.