The League Two outfit were much the better team in the first half and took a deserved lead through Padraig Amond's 38th-minute header from Robbie Willmott's cross.
Amond also hit the post before the break as County rattled Spurs, who relied on a set-piece goal to draw them level on 82 minutes as Kane tapped in at the back post.
While Newport may have missed their best opportunity to cause a famous giant killing, they do have the small matter of a replay at Wembley as a consolation prize.
Relive how the action unfolded in Wales with our minute-by-minute updates below.
4.30pmTo slightly paraphrase a famous David Brent quote, THIS is the real FA Cup. Eight-time winners and Premier League giants Tottenham Hotspur travel to League Two minnows Newport County. Can you smell the magic? Team news coming up...
4.34pmTOTTENHAM: Vorm; Trippier, Foyth, Vertonghen, Walker-Peters; Dier, Wanyama, Sissoko, Dembele; Llorente, Kane Subs: Gazzaniga, Sanchez, Davies, Georgiou, Amos, Dele, Son
4.36pmIt seems that both teams may have even rotated their social media officers for this match, and the stand-ins are being a bit slow with the team news. We will fill in those gaps for you as soon as possible, but what we can tell you is that Harry Kane starts and captains Tottenham this evening.
4.38pmFernando Llorente starts up front alongside Kane as Mauricio Pochettino makes five changes to the team which drew 1-1 with Southampton last weekend. Young defenders Juan Foyth and Kyle Walker-Peters are given another opportunity to impress, while Kieran Trippier and Victor Wanyama also come in.
4.40pmMichel Vorm is Tottenham's designated domestic cup keeper and continues in goal, having deputised for Hugo Lloris (illness) at St Mary's Stadium. The Dutchman is out of contract at the end of the season so will be looking to put in a solid performance to put himself in the shop window.
4.42pmSerge Aurier, Davinson Sanchez, Ben Davies, Dele Alli and Son Heung-min are all rested, while Christian Eriksen is not brought back into the starting XI after recovering from the same bug that his teammate Lloris caught while on a mid-season training camp in Barcelona last week.
4.44pmPochettino has named a very physical midfield, perhaps anticipating a combative approach from their lower-league hosts. There is also some concerns about the Rodney Parade pitch, which hosted a rugby match yesterday, so it may not be suited to the likes of Eriksen and Son.
4.46pmEric Dier's inclusion in the lineup gives Pochettino the option of using a back three, which might be how Spurs set up as otherwise it would be a very narrow midfield. Alternatively, Mousa Dembele may be asked to play off the left flank in an old-fashioned 4-4-2 formation.
4.50pmAs you can see, the town of Newport is buzzing for Tottenham's visit, and the man they are looking to to inspire a famous triumph is manager Mike Flynn, who has made two changes from the side which drew 1-1 with Morecambe last time out with captain Joss Labadie and Ben White recalled.
4.52pmJosh Sheehan and Mark O'Brien are the unlucky Newport duo who miss out on facing some of their Premier League heroes. Flynn names two up front - top scorer Padraig Amond and former West Ham United striker Frank Nouble, who have combined for 16 League Two goals this term.
4.54pmFinally we have confirmation of the substitutes, with Pochettino naming Dele Alli and Son in reserve and also young midfielders Luke Amos and Antony Georgiou. Dropped County pair Sheehan and O'Brien are at least given the consolation price of being on the bench.
4.56pmThere was talk that Toby Alderweireld may have been among the Spurs substitutes today after three months out with a hamstring injury, but there is no sign of the Belgian, or Erik Lamela who has suffered a fresh injury setback with a muscle strain. Eriksen and Lloris are given the day off.
4.58pmLamela has been linked with a return to Italy with Inter Milan in recent days, but his absence today is believed to be fitness related. The Argentine will be entering the last two years of his contract come the summer though, so there may be some substance to those Inter rumours.
5.00pmThe Argentine's career in North London could be further jeopardised by the arrival of Lucas Moura, who is poised to join from Paris Saint-Germain on a four-year contract, according to multiple reports this weekend. The Brazilian will be gutted to miss out on a dream debut at Rodney Parade.
5.02pmAll kidding aside, Spurs should not take this Newport side lightly. They were minutes away from being knocked out by League Two opposition at this stage last season when they had to score twice in the final few minutes to beat Wycombe Wanderers 4-3 in a thrilling tie at White Hart Lane.
5.04pmThe Welsh outfit are going pretty well in the league too. They are unbeaten in their last five matches and sit ninth in the table, just a single point adrift of the playoff places. Not bad four a side who were in the Conference four years ago, and only just survived relegation last term.
5.06pmThis is the first time since 1979 that Newport have reached the fourth round of the FA Cup. Spurs were semi-finalists last year - losing to Chelsea - but it was their best run in the competition in five years.
5.08pmRemarkably, it has been 27 years since the last time Spurs even reached the final of this competition. The 1991 victory over Nottingham Forest was also the most recent of Tottenham's eight FA Cup triumphs, which was once a record number. Their supporters will be thinking it is about time that wait ended.
5.10pmPochettino has previously made no secret of the fact that he sees the domestic cups as low priority compared to the Premier League and Champions League, but his inclusion of Kane today shows he is taking this game seriously. Perhaps Man City's form in the league has changed his mind...
5.12pmSTAT: Tottenham have progressed in each of their last 15 FA Cup ties as a top-flight side against teams from the fourth tier or lower, last being knocked out in January 1915 to non-league Norwich City.
5.14pmMeanwhile, Newport haven't won a FA Cup match against a top-flight opponent since January 1964, when they beat Sheffield Wednesday 3-2 in a third-round match at home.
5.16pmIn fairness, they haven't had many opportunities to play the big boys in recent seasons. The last time that Newport hosted a top-flight side was in October 1986 - a League Cup match against Everton in which they were on the end of a 5-1 second-leg battering.
5.18pmRodney Parade is filling up. It is not exactly Wembley but there will be a great atmosphere today as the 7,000 home fans aim to make Spurs uncomfortable.
5.20pmManager Mike Flynn has encouraged his players to believe that they can cause an upset. He has made it quite clear that his side aren't just in this tie to stand in awe of their opponents.
He told BBC Sport: "I go into every game, whether it be as a manager or a player, with a lot of confidence.
"And if any player thinks that we can not win on Saturday then they will probably be sat next to me on the bench or up in the stand because we have to believe we can cause an upset. Otherwise we may as well take the money from the tickets and give Tottenham the tie."
5.22pmMeanwhile, Spurs defender Jan Vertonghen has insisted that they won't take Newport for granted.
He told BBC Sport: "We've got a very professional team that will approach this game against Newport in a very good way. We just have to approach it in the way we approach Premier League games, which we definitely will."
5.24pmJust over five minutes until kickoff here, and it is clear why there were pre-match concerns about the pitch at Rodney Parade. The playing surface is cut up and quite boggy so it might be difficult for Tottenham to play their possession football. Going direct to Llorente might be their best option.
5.26pmThis is the first time in 58 years that Tottenham have played Newport, a year before their famous double-winning campaign. Club legends such as Danny Blanchflower and John White played that day as Spurs put four past the Exiles in an FA Cup tie.
5.28pmPREDICTION: The surroundings means this won't be easy for Spurs but I think they will sneak past Newport with a 2-1 victory. With the powerful midfield that Pochettino has named, the visitors don't have too many goal threats out there but Kane will fire them through.
1 minKICKOFF! The action gets underway. A proper FA Cup tie this - really looking forward to it.
3 minCHANCE! Brilliant chance for Newport to take an early lead as Dier is bundled off the ball on the edge of the own box. It is squared to Nouble who is in acres of space, but the former West Ham striker blazes a first-time shot way over the crossbar from 15 yards.
5 minSHOT! Labadie is too strong for Wanyama in the middle of the park, picking the ball off the recently fit-again midfielder and unleashing a 25-yard bullet which is straight into the hands of Vorm.
7 minIt has been a really uncomfortable start for Spurs, with Newport winning every 50-50 challenge to get the 10,000 strong crowd - extra seats have been put in specially for this tie - on their feet.
9 minCHANCE! Another decent chance for the hosts as Tozer launches a Rory Delap long throw into the box which is flicked on by Demetriou and turned goalwards by Bennett, but a few yards wide.
11 minTottenham have lined up three at the back, with Dier the centre of three defenders having returned to midfield in recent weeks. That means Kyle Walker-Peters has a big and difficult job getting forward down the left, which is not his preferred side, and providing service to Kane and Llorente.
12 minThe visitors advance into the final third for the first time of any note, with Llorente teeing up Kane for a 25-yard shot which flies into the stands. Chants of 'who are ya, who are ya', come from the Newport fans who are loving this start that they have seen from their team.
14 minThe ball drops to Bennett 25 yards out but his volley has all of the sting taken out of it by a deflection off Dier.
16 minTottenham are one of the finest architechts of the high press but they are getting a taste of their own medicine tonight. Newport are putting a lot of pressure on the Spurs centre-backs and the likes of Foyth and Vertonghen need to be very careful.
18 minBeneath their feet, Spurs have a bobbly pitch to contend with and, above their heads the visitors have the wind against them in the first half. Newport have knocked a couple of balls over the top which have been taken by the strong breeze and caused Spurs a few problems.
20 minTottenham were just beginning to get a flowing move going but Trippier overcooks a pass meant for Sissoko in the right channel. Spurs have not had much possession at all in the Newport half.
22 minNewport are winning all of the aerial battles so far. Midway through the first half and Mike Flynn will be very happy with how his side have unsettled Spurs, and created a few good chances too.
24 minSAVE! Superb handling from Vorm as Sissoko gets an inadvertent flick on a Willmott free kick at the near post, which could have easily caught out the Dutch goalkeeper but he reacted very well.
26 minThat Tozer throw-in is a real weapon for Newport but a push on Wanyama in the box allows Tottenham to escape with a free kick.
28 minTottenham are enjoying their best spell of possession so far but have still yet to create a decent chance. Walker-Peters cuts inside onto his right foot 20 yards from goal but Butler gets a block in.
31 minWOODWORK! Finally Kane has a sniff of a chance and he nearly puts Spurs ahead. Dembele spins away from his man beautifully in midfield and starts an attack which eventually releases Sissoko, whose cutback is turned onto the near post by the prolific Spurs striker.
32 minYELLOW! Exiles captain Labadie becomes the first player to be carded this evening for a foul on Dembele. The midfielder served a couple of bans for biting earlier in his career so can be a bit of a wildcard. He's started this game very well though.
34 minFrom the resulting free kick, Kane shoots a few yards over the crossbar. Not too far away, but never troubling Newport keeper Day.
36 minLabadie skilfully juggles the ball away from Vertonghen and Walker-Peters and fizzes a cross towards the penalty spot. Dier is on hand to cut the ball out at the near post, at the cost of another Tozer throw.
38 minNewport take the lead, and you can't say that they haven't deserved it! Willmott stands up an inviting cross to the back post, and Amond gets above Trippier to head down and past Vorm into the bottom corner. Rodney Parade goes wild and we have an FA Cup giant-killing on our hands here.
40 minPochettino looks equal parts concerned and stumped on the Tottenham bench. He named a side suited to a physical contest, but they have been outbattled by Newport and are lacking the creativity to test the County defence much themselves.
42 minWOODWORK! Tottenham are well and truly spooked here, with Dier scuffing a clearance to Amond in the area and the goalscorer striking the post from a very tight angle. Vorm got a touch to it and Newport have a corner.
44 minTrippier's delivery from a wide free kick is into a really good area but Demetriou clears for Newport.
45+1 minThere will be two minutes of stoppage time at the end of this half. Ironic cheers greet a piece of miscontrol from Walker-Peters as the ball goes through his legs and out of play for a throw-in, which pretty much sums up Tottenham's half. The youngster far from the biggest culprit, mind you.
6.20pmThe half-time whistle is greeted with an enormous cheer by the Newport fans, who should be very proud of their team after that first-half display. They unsettled Spurs and created a succession of good chances, taking one in the 38th minute when Padraig Amond popped up at the back post.
6.22pmAll credit should go to Newport, who have been first to every second ball, won all of the 50-50 challenges and put really good pressure on the Tottenham defenders. They have spooked Spurs and it will take a strong response from the visitors for them to get themselves out of this hole.
6.24pmPochettino has got this badly wrong, from a tactical perspective. The wing-backs have not been able to get forward and that is the only way that Spurs are going to get service to their front two with the lack of a creative player in midfield. Changes need to be made at half time.
6.26pmDele Alli and Son are both options on the bench that Pochettino needs to consider. The Argentine doesn't make many early chances, but that first-half performance will force him into action. Ben Davies is also an option if Spurs stick with wing-backs as Walker-Peters is uneffective on the left.
6.28pmSTATS: Tottenham failed to muster a single shot on target in that half despite having over 60% of possession, although Harry Kane did hit the post with the scores at 0-0. That was Tottenham's only chance, and there has been very little link up between Kane and Llorente.
6.30pmThe entire Newport team has been excellent, but a special word of praise for their captain Joss Labadie who has led their high press superbly and bullied some powerful and experienced players in that Tottenham midfield.
46 minKICKOFF: The second half is underway with Spurs making a change, bringing on Son in place of Walker-Peters. Looks like Vertonghen has gone to left-back in a 4-4-2 system.
48 minTottenham have started this second half in control of possession. Pochettino will want his players to show their quality in the second half, but also a bit more fight and aggression in midfield where Wanyama and Dembele have been bullied without so much as putting a tackle in.
50 minSpurs have a decent opening but Kane takes too long in shifting the ball out to Son and Newport can get some defenders back. The striker appeared to be weighing up whether to shoot or pass, and by the time that he made his mind up it was too late.
52 minKane tries to work some space for a shot on the edge of the box but Bennett stands up strongly and forces him away from goal.
54 minTozer has the wind to assist him with his long throw in this half. This is proper throwback stuff to the Rory Delap days. There will be Arsenal defenders from the late 2000s hiding behind the sofa watching this, although they'd also be pretty happy with the current scoreline.
56 minEleven minutes into the second half and it has been slightly better from Spurs but still Day remains untested in the Newport goal.
58 minMISS! Trippier whips in a fabulous cross and Kane heads over the crossbar from four yards out. What a chance, and it may have been an easier one for Dier had Kane left it for the defender behind him.
60 minSissoko has had a couple of Sissoko moments in the last couple of minutes. He has his assets in a central midfield role but on the right wing he looks completely out of place. Lucas Moura shouldn't fear for his place in the team based on this performance from the Frenchman.
62 minWanyama makes a decent run into the box but the Tottenham attack breaks down at the Newport wall.
63 minSpurs work the ball to Kane in the box but he cannot quite get his feet sorted and White intervenes.
65 minSUBSTITUTION: Dele Alli is on in place of Llorente, who has once again failed to impress but it is hard to place any blame at the Spaniard's feet, or head, as he simply hasn't had the type of service that he can thrive on.
67 minSAVE! Day has not been anywhere near as busy as he may have expected to be, but makes a crucial save when he is called upon here. Alli plays the ball into the path of Son, whose curling effort is blocked by the legs of the Newport goalkeeper.
69 minA sign of desperation from Spurs as Foyth takes aim from range and shots high and wide with his left foot.
71 minSHOT! Kane spins away from his man on the left edge of the area and shoots into the side-netting, and it looked as though Day had it covered. I'd be getting nervous now if I was a Spurs fan. Oh wait, I am. Bugger.
73 minSUBSTITUTION: County boss Mike Flynn makes his first change, bringing off goalscorer Amond for Shawn McCoulsky, the Bristol City loanee who scored the winner against Leeds United in the last round.
75 minAs for Pochettino, he has already played his Son and Alli cards and does not have many other options to turn to on the bench, as he is unlikely to turn to either Amos or Georgiou in this situation. Davies may give Spurs a more natural attacking outlet down the left so may be called upon.
77 minLabadie is perhaps a little lucky to escape without a second yellow card after cycnically tugging back Dembele.
79 minDemetriou makes an important tackle on Son in the box and gets the fist pump treatment from his teammates. Newport have remained very focussed and have a little over 10 minutes to see out.
80 minTottenham can't get in behind the Newport defence and a plodding series of passes ends with Wanyama slicing an ugly shot wide. The pitch has not been conducive to passing football but Spurs cannot use that as an excuse for creating so little.
82 minHeartbreak for Newport and relief for Spurs as Kane bags them an equaliser. Son flicks on a corner at the near post and the Spurs saviour is left unmarked in the six-yard box for a simple tap-in.
84 minSUBSTITUTION: Labadie is clapped off after a proper captain's performance, with Matt Dolan on to provide fresh legs in midfield. Newport will surely take a lucrative Wembley replay now.
86 minSpurs are pushing for a winner as Trippier knocks a good ball into the right channel for Son. He squares up his marker but Butler gets a block in to see Newport escape with a corner.
88 minSUBSTITUTION: Foyth collects a yellow card for wiping out Tozer on the right wing. Tottenham could have done with a few tackles like that in the first half but have improved aplenty since the break.
90 minNewport win a corner which Vorm collects at the second attempt after a slight wind-induced wobble. There will be four minutes of stoppage time. Can County survive or will Spurs pinch a winner?
90+2 minSon is dispossessed on the halfway line but McCoulsky does not have the support with him to spring on Tottenham.
90+3 minGreat defending from White who blocks a close-range prod from Kane after he had skilfully lobbed the ball over Demetriou. That might have been the chance that Spurs were waiting for.
90+4 minKane whips in a fantastic cross which Alli comes within inches of making contact with. Day had rushed off his line and would have been helpess had the midfielder got his head to the ball.
7.28pmNo famous cup giant killing, today at least but Newport will get a second chance at Wembley in a couple of weeks. That lucrative replay may actually be a better result for the League Two side in the long run, and the least they deserved after giving Tottenham a real scare today.
Spurs will be relieved but this is a replay they could have done without ahead of a crucial run of upcoming Premier League fixtures.