The Scot took the reins at Carrow Road in January 2015, leading the Canaries to the top flight that season, although a disappointing campaign saw them relegated straight back to the Championship.
Despite a strong start to this season, Neil's side have now suffered five straight league defeats to drop out of the playoffs and sit 13 points adrift of league leaders Newcastle United.
"We have come from a more difficult position to get promotion. I came in on January 9 that season and it'll be interesting to see where we sit between now and then," Neil told The Pink Un.
"It is not as if we are where Aston Villa were earlier in the season, third or fourth from bottom. I just think it is expectation levels, which is fair, but we have long way to go.
"Goals conceded and goals scored are big statistics people look at. However, if you get promoted at the end having conceded a lot of goals you would not be fussed about the statistics. As far as we are concerned, we have conceded far too many. We are far too good to have conceded that many and it is something we have to improve on if we want to be at the top end."
Neil's side will look to get back to winning ways when they welcome Brentford to Carrow Road on Saturday.