EU regulators have blocked the tie-up of exchange operators Deutsche Boerse and NYSE Euronext to stop them taking a stranglehold on the European futures market.

Deutsche Boerse and NYSE Euronext, which unveiled the $7.4 billion plan to create the world's biggest stock exchange as long ago as February last year, said they would unwind the deal, the sector's fourth failed tie-up attempt in a year.

The European Commission consulted more than 700 market participants and stakeholders during its assessment and said in a 459-page document on Wednesday that the combined entity would make it hard for new players to compete.

The merger between Deutsche Boerse and NYSE Euronext would have led to a near-monopoly in European financial derivatives worldwide, EU Competition Commissioner Joaquin Almunia said in a statement.

These markets are at the heart of the financial system, and it is crucial for the whole European economy that they remain competitive. We tried to find a solution, but the remedies offered fell far short of resolving the concerns.

Deutsche Boerse and NYSE Euronext had refused to sell either the German operator's Eurex derivatives market operator or the U.S. company's London-based futures exchange Liffe to address such concerns.

END OF MEGA-DEALS?

Analysts said the regulatory veto might put a temporary brake on the next wave of deals.

The motivation to do deals is as relevant as ever, with exchanges needing to scale up and seek cost synergies, but the failed deals over the past year will make investors and companies more cautious, said Richard Perrott, analyst at Berenberg Bank.

Deutsche Boerse/NYSE Euronext was an unusual deal in that there were obvious competition concerns that wouldn't exist with most other exchange combinations.

NYSE Euronext Chief Executive Duncan Niederauer told CNBC he expected bolt-on acquisitions in the next wave of consolidation in the industry rather than mega-deals.

Clearing house LCH.Clearnet, now in takeover talks with the London Stock Exchange, and the London Metal Exchange could be on NYSE Euronext's radar, said Peter Lenardos, analyst at RBC Capital Markets.

Deutsche Boerse doesn't need to do a deal, while I'd suggest it is no coincidence the NYSE Euronext CEO is refusing to comment on LCH and the LME, because these are the types of deals he should now be looking at, he said.

Nasdaq OMX Group Inc Chief Executive Bob Greifeld said large deals are still feasible, but companies would need to take into account the regulator's technical analysis.

Now, when you look at this, any time you are trying to complete a merger where the number of competitors is low or the market share is high, your success will hinge on whether you can broaden the definition of the market, he told a conference call after Nasdaq posted better-than-expected quarterly earnings.

NYSE Euronext said it would return $550 million to shareholders via a share repurchase programme and seek to grow its derivatives business.

We will also take advantage of our financial strength to capture opportunities for growth in derivatives, and through our new initiatives, including technology services, NYSE Liffe US/NYPC and post-trade services, CEO Niederauer said.

Deutsche Boerse shares were up 0.8 percent in afternoon trade, while German blue-chips were up 2 percent.

NYSE Euronext shares were also up 0.8 percent in New York.

U.S. regulators had approved the merger in December last year on condition the exchanges sell a minor asset.

In the past year the sector has seen three other large deals fail.

Nasdaq and IntercontinentalExchange Inc's bid for NYSE Euronext was rejected by the U.S. Department of Justice, London Stock Exchange's takeover of TMX Group was rejected by shareholders of the Toronto Stock Exchange operator, and Singapore Exchange Ltd's bid for Australia's ASX Ltd was stopped by the Australian government.

(Additional reporting by Edward Taylor and Christoph Steitz in Frankfurt, Luke Jeffs in London, John McCrank in New York, editing by Sebastian Moffett and Will Waterman)