Following the United Kingdom’s decision to exit the European Union, there has been an immense amount of speculation over how the economy — on both sides — will handle this change. According to the British Bankers’ Association (BBA), U.K. has a big reason to worry — major banks are looking to move out of the country early next year over the uncertainty surrounding Brexit.

“Banking is probably more affected by Brexit than any other sector of the economy, both in the degree of impact and the scale of the implications,” head of the BBA, Anthony Browne, wrote in a column for the Guardian on Saturday.

Before the June referendum, the BBA had conducted a survey that showed that a large majority of banks wanted to remain in the Union and Browne, who believes that even smaller banks could move out by 2017, wrote: “Their hands are quivering over the relocate button.”

Browne, who made similar comments at the BBA annual conference last week, attributed some of the tensions in the banking sector to the current “public and political debate,” which he said was taking the banks in a “wrong direction.”

At the conference last week, he said banks had already “set up project teams to work out what operations they need to move by when, and how best to do it,” the BBC reported.

“It is the U.K.’s biggest export industry by far and is more internationally mobile than most. But it also gets its rules and legal rights to serve its customers cross-border from the EU,” Browne explained. “For banks, Brexit does not simply mean additional tariffs being imposed on trade — as is likely to be the case with other sectors. It is about whether banks have the legal right to provide services. ”

While banks are calling for transition arrangements to be made after the formal exit of the U.K., Browne warned that the mood was not conducive to an easy way out with both sides — the EU and those who wanted to leave it in the first place — skeptical of each other.

“The problem comes — as seems increasingly likely, judging by the rhetoric - when national governments try to use the EU exit negotiations to build walls across the Channel to split Europe's integrated financial market in two, in order to force jobs from London,” Browne said.

“From a European perspective, this would be cutting off its nose to spite its face. It might lead to a few jobs moving to Paris or Frankfurt but it will make it more expensive for companies in France and Germany to raise money for investment, slowing the wider economy.”

Brown concluded by affirming that while there was a way out but repercussions would be severe for both sides. He explained, “London will survive as a global financial centre. Finance is inventive and will find a way through. But putting up barriers to the trade in financial services across the Channel will make us all worse off, not just in the UK but in mainland Europe.”