Stocks fell on Wednesday as investors worried the Federal Reserve is closer to pulling back on extraordinary measures to inject funding to shore up the economy.

The Fed's policy-setters met and kept interest rates unchanged, as expected, but they also said the U.S. central bank would slow purchases of mortgage debt to extend that program's life until the end of March. That was seen as a step toward a measured withdrawal of its extraordinary support for the economy during the downturn.

They're talking about removing some of the various packages they have in place for purchasing mortgages and other instruments in debt markets that was kind of keeping everything flowing, said Kurt Brunner, portfolio manager at the Swarthmore Group in Philadelphia.

There's a broad concern about what happens when the Fed gets out of the way.

Among the casualties were banks, housing stocks and energy shares. The Dow Jones home construction index <.DJUSHB> slid 3.4 percent, while the S&P energy index <.GSPE> declined 2 percent, a decline that coincided with a sharp slide in crude oil prices.

The Dow Jones industrial average <.DJI> shed 81.32 points, or 0.83 percent, to 9,748.55. The Standard & Poor's 500 Index <.SPX> declined 10.79 points, or 1.01 percent, to 1,060.87. The Nasdaq Composite Index <.IXIC> lost 14.88 points, or 0.69 percent, to 2,131.42.

Initially stocks had risen sharply following the Fed's comment that economic activity was picking up, but in the last hour of trading the market reversed course as investors fretted about the timing of the removal of some of the Fed's stimulus.

The other worry was the Fed's vow that interest rates will stay low for an extended time.

Keeping interest rates low -- it's positive for the consumer, but it's tougher for banks to make money with interest rates so low, said Dan Faretta, senior market strategist at Lind-Waldock, a brokerage firm in Chicago.

The market's run-up of nearly 60 percent over six months might have also caused some investors to use the Fed's statement as reason to book profits, he added.

Slowing mortgage purchases is something the Fed could consider in another six months, Faretta said. There's still a lot of problems with mortgages, the housing market in general, as well as the banking sector.

Among banks, JPMorgan fell 3.03 percent to $45.06, making the stock the Dow's top drag. The S&P financial index <.GSPF> and the KBW bank index <.BKX> each lost 2.1 percent.

Home builder Toll Brothers fell 3.4 percent to $20.68. In energy, Chevron Corp fell 1.7 percent to $71.73, while U.S. front-month crude futures declined $2.79 or 3.89 percent to settle at $68.97 a barrel.

Networking equipment maker Cisco Systems weighed heavily on Nasdaq, falling 2.6 percent to $22.80.

Volume was moderate, with about 1.32 billion shares changing hands on the New York Stock Exchange, compared with last year's estimated daily average of 1.49 billion. On the Nasdaq, about 2.72 billion shares traded, above last year's daily average of 2.28 billion.

Declining stocks outnumbered advancing ones on the NYSE by a ratio of about 9 to 5, while on the Nasdaq, about eight stocks fell for every five that rose.

(Additional reporting by Leah Schnurr; Editing by Kenneth Barry)