The 20-minute Flash Crash on May 6, 2010, sent the Dow Jones Industrial Average sinking 600 points. Reuters/Lucas Jackson

At 2:42 p.m. on Thursday, May 6, 2010, U.S. financial markets convulsed. The Dow Jones Industrial Average, already 300 points down from the start of the day, plummeted 600 more in a matter of minutes. Liquidity evaporated, prices fell precipitously and chaos ensued.

Some stocks traded for a penny apiece, others for $100,000. Within 20 minutes prices had rebounded and the “flash crash” was over. But confidence in U.S. financial markets was severely shaken.

All eyes fell on the Securities and Exchange Commission and Commodity Future Trading Commission as the agencies embarked on a five-month study into the causes of the crash. But even before the results were in, it was clear that the fragmented markets needed more complete oversight, a system that would track every offer, trade, cancellation and modification entered on every asset in real time.

Mary Schapiro, then-chairwoman of the SEC, promised the system would be “invaluable to enhance the ability to detect and monitor aberrant and illegal activity across multiple markets.”

But as the SEC and CFTC tinkered toward remedies, the fishy trading activities that regulators now say helped tank the market that day in 2010 have continued unabated, says Eric Hunsader of the market analysis firm Nanex.

“A hundred weird things happen every day,” says Hunsader, whose company provided data to the CFTC for its Flash Crash investigation and continues to regularly publish reports on alleged instances of spoofing, layering and other types of market manipulation. “They’ve become daily occurrences.”

Hunsader points to irregularities he detected in gold futures just last week. On April 28, Hunsader wrote a post purporting to show traders entering large volumes of orders in gold markets with no intent to carry them out, a violation of market rules.

Three days later, the Chicago Mercantile Exchange announced disciplinary actions against two traders accused of engaging in the market manipulation Hunsader had described. (Hunsader says Nanex.net traffic data show CME computers visiting his site April 28.)

Hunsader isn’t the only one concerned about the state of market surveillance. Five years after the flash crash sent financial markets reeling, experts see near-insurmountable challenges standing between fundamental reforms and their implementation.

The most significant regulatory effort proposed in the wake of the flash crash is nowhere near finished. The consolidated audit trail (CAT) would gather real-time information on every trade and order across 19 national securities exchanges, giving regulators a comprehensive look into exactly who is trading what at all times.

Though the Flash Crash took only minutes to transpire, the CAT could end up taking more than 10 years to come into effect, the result of regulatory complexity, legal wrangling and mounting technical headaches. TABB Group, a market research firm, says the CAT won’t likely be ready until 2020 at the earliest.

“The existing surveillance monitoring paradigm was incapable of understanding what had happened,” says TABB Group’s Alex Tabb, who has followed the implementation of the audit trail. “The CAT is a huge undertaking.”

Joe Saluzzi, a co-founder of Themis Trading, in New Jersey, says the SEC and CFTC have made progress, but “nothing really earth-shattering” has been implemented.

“It’s a slow slog, and they’ve got a lot of things on their plate,” Saluzzi says.

Enter The Spoofer

Last month, the U.S. Department of Justice stunned the financial community by announcing charges against British trader Navinder Singh Sarao, who allegedly made more than $40 million by spoofing the market for Standard & Poor's 500 futures called E-Minis -- including on the day regulators say E-Minis precipitated the flash crash.

The new narrative seemed to clash with the original Flash Crash report, which said that a large order from a Kansas trading firm helped spark a chain reaction that was exacerbated by high-frequency traders. The final analysis said little about spoofing and saw algorithmic traders as contributing to the Flash Crash, not causing it.

As Sarao challenges extradition to the U.S., the case against him raises more questions than it answers.

“The spoofing case is perfect evidence that they don’t have any idea what’s going on during the day,” Saluzzi says. “A guy like Sarao can come in and spoof and spoof and spoof, and nobody sees him.” The Justice Department said Sarao engaged in spoofing well into 2014.

With a consolidated audit trail, Sarao’s alleged manipulations would have been noticeable sooner. In the existing framework, regulators spot malfeasance then request the identifying data from exchanges to track down an individual. The CAT would allow for near-instant identification.

But that doesn’t mean regulators have been asleep at the switch. The SEC in 2010 banned stub quotes, offers to trade stocks at a penny or $100,000 that served no legitimate purpose but ended up being executed during the Flash Crash.

Regulators also implemented so-called “circuit breakers,” which freeze trading on stocks that deviate suddenly, helping to stave off the sudden dislocations that marked the Flash Crash.

Saluzzi, however, says the SEC’s strongest efforts have been in pushing the private companies that run public exchanges, such as Nasdaq and the CME, to tighten their own rules. The CME, for instance, announced stricter oversight around spoofing in September, and has since increased its disciplinary referrals.

But for Saluzzi and others, the central problem is deeper, and less manageable: the hopeless complexity of the markets. Trading of stocks and options occurs over more than a dozen “lit” exchanges and several dozen more anonymous “dark pools,” each of which has its own reporting standards. The SEC and CFTC split the task of regulating equities and futures, respectively, placing what some see as an artificial division between deeply entangled markets.

That complexity has made implementing the SEC’s landmark CAT program a herculean effort. The SEC delegated the implementation of the initiative to the participating exchanges, and six companies are still in the running for a contract to design the system.

“There’s no one entity who is completely responsible” for the audit trail, says Shagun Bali, a researcher at the TABB Group. When it comes to the SEC, she says, “they’re responsible for it but they don’t want to take responsibility.”

With each exchange guarding its own proprietary interests, the CAT process has dragged on for five years with no end in sight. Tabb compares the squabbling among self-regulating exchanges to the European Union: “It’s like a whole bunch of independent nations with their competing interests at play.”

It wouldn’t be the first time that privately managed exchanges have put up roadblocks to regulatory efforts. In 2012, lawyers representing the CME demanded that the SEC shutter a promising research program that the exchange said put its clients’ identities at risk. The SEC quickly complied.

Oversight, Under Budget

The question remains how the SEC will fund the audit trail, which could increase costs for small-time broker-dealers.

The CFTC has been even harder-pressed for funding. In comments last year, CFTC Commissioner Scott O’Malia warned that planned budget levels shortchanged the agency on crucial technological investments.

“The commission will waste another year without deploying critical technology, such as an order message data collection and analysis system, a key tool for surveillance,” O’Malia wrote.

This year, CFTC Chairman Timothy Massad requested an additional $28 million for technical infrastructure, warning that without better funding the CFTC would be “severely limited in its ability to detect fraud and manipulation, market abuses, firms in trouble, or other improper behavior, thereby significantly increasing the potential costs and risks to our markets and our financial system generally.”

With real-time surveillance nonexistent and financial markets as fragmented as ever, few are confident that regulators can prevent another meltdown like the Flash Crash. Subsequent minicrashes in U.S. Treasury futures and during Facebook's initial public offering have kept investors on edge.

"The exit door is very small,” says Hunsader, recalling a 2013 hack of the Associated Press Twitter account that briefly jolted the S&P 500. “A fake tweet almost got us there.”

Tabb is hopeful that regulators and exchanges will overcome technological hurdles, but he says that there’s no undoing the complex knot that markets have become. “Humpty Dumpty has fallen. And you can’t put Humpty Dumpty back together again.”