The 21-year-old, whose best result thus far is a 12th-place finish, was given a five-place grid penalty at the Bahrain Grand Prix last time out for crashing into the back of Adrian Sutil.
"A lot has to do with the performance of the car, so you have to be careful not to start making assessments after four races, but yes, we have to keep a very close eye on him and see if the expectations are satisfied or not," Kaltenborn told AutoSport.
"We have no doubts about his mental capabilities to get on top of things and understand them. But he is struggling at the moment with his performance. So we need to try to focus on get that potential out because we know it's there.
"The consistency is not there now. Of course we can say that when you hire a rookie driver you can't expect that at this stage. I think after four races it's fast too early to make that kind of assessment. You can't expect too much, on the one side, but on the other you have to expect a lot because you just have limited opportunities. And it's very difficult to balance this out."
Gutierrez finished third in the GP2 last season.