Spurs are not expected to move into their new stadium until 2017-18, with the club facing a season away from home while construction work is carried out.
The Hammers are due to move into the Olympic Stadium in August 2016, and Brady has admitted that they would be unlikely to give permission for Spurs to share with them.
"In reality they probably could – but only with our permission," The Guardian quotes Brady as saying.
"No-one has asked us for our permission and, if they did, we would probably say no, depending on who it is – if you get my drift.
"We are the anchor tenant for the winter matches and nothing else can happen in that time without our permission and our football matches take priority over everything else."
West Ham return to Premier League action with a trip to Manchester United on Saturday.