Top bankers have warned that Brexit could force their industry to shift jobs out the City, as a senior policymaker at the Bank of England warned the loss of such high-paid roles could knock a hole in the UK’s budget.

Anil Kashyap, a newly appointed figure at the Bank, said the UK would lose out from taxes paid by City workers if jobs moved out of the capital as a result of a hard Brexit.

Appearing before MPs on the Treasury select committee, Kashyap was asked about his expectations for the value of the pound if the UK were to exit the EU quickly and without access to the single market. The pound has sunk 15% since the 23 June referendum.

Citing figures that 3.7% of UK employment is in financial services, which provides 11.5% of the country’s tax take, Kashyap said: “You don’t have to lose much of that before you cause a hole in the budget that could be meaningful and that, I think, would have some knock on effects for the exchange rate.

“The one point I haven’t seen stressed as much in this debate is just the pure fiscal cost to the budget. If you’re going to lose, then, a bunch of financial services jobs that are going to migrate back, the fiscal consequences of that is non-trivial, just because those jobs are so high paying and [they pay] so much tax.”

Kayshap, an academic at the University of Chicago, has been appointed to the Bank’s financial policy committee, which was set up shortly after the 2008 banking crisis to look for risks to financial system.

He gave his evidence to the MPs as London-based bankers of major US financial firms outlined the implications of Brexit if the UK did not secure a deal to retain access to the remaining members of the EU.

Rob Rooney, head of UK operations at Morgan Stanley, told a conference that operations would have to shift out of the country. “It really isn’t terribly complicated. If we are outside the EU and we don’t have what would be a stable and long-term commitment to access the single market then a lot of the things we do today in London, we’d have to do inside the EU 27,” he said in quotes reported by Reuters.

James Bardrick, who runs the UK operations of US bank Citi, said decisions about moving roles could be made next year – when Theresa May suggested she would trigger article 50 and formally begin the process of exiting the EU.

“How do we and when do we start making decisions … knowing the plan is ready to go … it could be in the first quarter of 2017,” Bardrick said according to Reuters.

The City minister, Simon Kirby, told the audience that “an absolute priority” for the government was keeping the UK’s as “one of the best financial centres”.

Some City firms have been concerned about the fate for employment in the EU , but Kirby said: “We fully expect that the legal rights of EU nationals already in the UK will be properly protected.”

Kashyap’s admission to MPs about the fiscal costs of Brexit followed a report in the Times that leaked government papers showed Treasury coffers could take a £66bn annual hit if there was a hard Brexit.

However, Downing Street distanced itself from the figures, saying they were not new. Meanwhile, a report by Oliver Wyman, for the lobby group CityUK, the UK’s financial services industry, has warned hard Brexit could put 35,000 jobs at risk.

Kashyap also told MPs about that weakness in banks across the EU could hurt major players in the UK. “I am particularly worried about contagion from stress coming from the European banks and whether there might be linkages – perhaps indirect – between them and the largest UK banks,” Kashyap said.

He did not expand on those comments in the hearing but said in written evidence: “For example, rumours about the health of various large European banks have been reported by the media on and off for the last few months … If any were to need a substantial recapitalisation is possible their lenders will reassess exposures to all their counterparties”.

Deutsche Bank’s shares have been pushed to 30-year lows on concerns that it could face a $14bn (£10.5bn) settlement with the US justice department over a decade-old mis-selling scandal. Kashyap said the penalties could appear “a little bit arbitrary”.