Big banks and trading firms that fail to settle certain mortgage-security trades will have to pay a penalty starting Wednesday as part of an effort to fix a persistent problem that adds risk to financial markets.

The penalty reflects growing concern that firms are struggling to complete billions of dollars of trades in the most unlikely of places: the world's most liquid markets.

A Reuters review of data from the U.S. Securities and Exchange Commission and the Federal Reserve Bank of New York shows a persistent failure of settlement in fixed-income and equity markets, including trading of exchange-traded funds, a popular investment for retail and institutional investors.

On an average trading day in 2011, failures to deliver shares amounted to 4.3 percent of the total volume of shares traded that day, according to data analyzed by Reuters.

That represents about $2.5 billion (1.5 billion pound) of stock value every day, more than the market capitalization of the Wendy's Co restaurant chain or theme park operator Six Flags Entertainment .

Similarly, in fixed-income trading by the largest dealers, anywhere between 6 percent and 11 percent of bonds -- or $52 billion to $93 billion worth -- did not get delivered on time, Reuters found.

Experts say settlement failures should be very close to zero, given that computerized trading systems allow buyers and sellers to match trades with great speed and efficiency.

We've got a fundamental structural problem, said Fred Sommers, a capital markets operations research consultant at Basis Point Group in Boston.

The inability to settle trades, known in the industry as a fail, occurs when a seller of securities fails to turn over a security on a date that seller and buyer agreed on.

The failure to settle trades can significantly heighten risk in the markets, create a loss of confidence between counterparties, and lead to a daisy-chain of uncompleted trades, according to some industry experts.

Fails can increase operational costs and counterparty credit risk, absorb scarce capital through regulatory charges, and damage customer relations, according to a report last year by the Treasury Market Practices Group, a committee of experts sponsored by the New York Fed.

The prospect of persistent settlement fails at a high level can cause market participants to temporarily withdraw from the market, or even exit the market, adversely affecting market liquidity and stability, the report said.

The Treasury Market Practices Group includes senior bankers from some of the world's biggest banks and securities firms, including Morgan Stanley , Goldman Sachs Group Inc , Barclays Plc , Citigroup Inc and JPMorgan Chase & Co .

The risk has prompted regulators, a U.S. congressional panel and industry groups sponsored by central banks to find ways to decrease the prevalence of failed trades.

One change takes effect Wednesday: Banks and trading firms will have to pay a penalty if they do not properly complete certain mortgage trades. A fee of 3 percent minus overnight borrowing costs will be charged for debt sold by Fannie Mae, Freddie Mac and the Federal Home Loan Banks. A fee of 2 percent minus overnight borrowing costs will apply for bonds underpinned by Fannie Mae, Freddie Mac and Ginnie Mae-backed loans

A similar fee was placed on U.S. Treasury bonds in 2009. While that helped lower the prevalence of fails in Treasury trades, problems persist in the settlement of other securities, particularly stocks and ETFs.

Experts say hedge funds and other firms that trade equities and ETFs often move in or out of securities in the final minutes of the trading day, leaving little time for trades to settle. ETFs hold more than $1 trillion in baskets of stocks or other assets often designed to mirror an index. The funds are popular with investors because they trade during the day and are cheaper than mutual funds.

ETF providers deny there is a substantial problem. They say market-makers can agree to take up to a week to complete a trade, but that may not show up in the statistics.

Much of the data on ETF settlement failures is inaccurate, said Jim Ross, head of State Street Global Advisors' ETF business.

Stock settlement failure data is compiled by National Securities Clearing Corp and published by the SEC. The SEC publishes only the cumulative number of shares not delivered on time as of each day, which may also include failed trades from previous days. National Securities Clearing Corp, through its parent company, declined to provide additional data to measure only the new fails occurring each trading day, so Reuters gauged overall trends in settlement fails by comparing each day separately against overall market activity. Reuters applied similar methodology to the fixed-income data from the Fed.

The problem of failing to properly settle U.S. Treasury security trades on time was identified as early as 2005 in a paper by the New York Fed, which highlighted periods when liquidity may be jeopardized by elevated and persistent fails.

The paper said that settlement fails occur for a variety of reasons, including miscommunication or computer problems between buyer and seller, whose operations departments might have different details for when a trade should close.

The paper said the fails could lead to a daisy chain of cascading fails: (Trader) A's failure to deliver bonds to B causes B to fail on a delivery of the same bonds to C, causing C to fail on a similar delivery to D, and so on.

Problems continued despite the concerns raised. In a move to fix the problem, the Treasury Market Practices Group in 2009 put in place a fails charge -- effectively a financial penalty -- to halt the failure of dealers to settle U.S. Treasury trades.

That charge led to a decrease in U.S. Treasury fails. The problems, though, moved to the trading of agency mortgage-securities, or bonds backed by home loans guaranteed by Fannie Mae and Freddie Mac.

In a study last year, the Ewing Marion Kauffman Foundation, which oversees assets of about $2 billion, said, Unfortunately, like squeezing a balloon, the fails problem has not gone away. It simply has moved to markets where fails are not punished.

One reason: Banks and trading firms use these securities as collateral for cash loans. In normal times, the borrower would invest the cash overnight and earn interest. But with bank borrowing rates at a low, there is little reason to actually deliver the securities to obtain the cash. There is also little incentive to finally deliver securities to avoid paying a low borrowing rate.

The new penalty in the agency mortgage market that goes into effect on Wednesday aims to provide that incentive.

You're essentially trying to increase the cost of failing, said Chris Killian, a managing director at the Securities Industry and Financial Markets Association, which represents banks and securities firms. They wanted to develop incentives to mitigate the fails.

(Reporting by Carrick Mollenkamp, Cezary Podkul and Jessica Toonkel; in New York; Editing By Alwyn Scott and John Wallace)