Wall Street's four-day rally ground to a halt on Thursday, with major indexes falling 1 percent on caution ahead of a key labor market report expected to underscore fears the economy is headed for another recession.

Financials were the biggest losers, selling off sharply in the afternoon, led by Goldman Sachs Group Inc . Goldman's shares fell 3.5 percent to $112.16 after agreements with the Federal Reserve and New York state's banking regulator over wrongful foreclosures raised concerns that Goldman is still not yet off the hook.

JPMorgan Chase & Co and Bank of America Corp were the two biggest losers on the Dow, both falling more than 3 percent.

A day ahead of the government's release of monthly payrolls data, a decline in the employment component of the Institute for Supply Management's factory activity index heightened worries that August jobs growth will be weaker than feared. ISM's factory activity index came in only just above the level that indicates growth.

Recent employment indicators suggest zero growth in private payrolls, said Jack Ablin, chief investment officer at Harris Private Bank in Chicago. If that comes to pass we are going to have some big disappointments tomorrow.

In another discouraging sign, the White House, already struggling to turn around the high U.S. unemployment rate, cut its economic growth outlook for the next two years.

After dropping more than 17 percent from early July to early August, the S&P 500 had risen by 9 percent heading into Thursday's session, leaving investors reluctant to place big bets a day ahead of the August labor report, which is expected to show an increase of 75,000 jobs.

The news over the past few days hasn't been conducive to the rally continuing, said Randy Bateman, chief investment officer of Huntington Asset Management in Columbus, Ohio.

Bateman, who helps oversee $14.5 billion, said he was not optimistic about the payroll report being strong.

The Dow Jones industrial average <.DJI> was down 119.96 points, or 1.03 percent, at 11,493.57. The Standard & Poor's 500 Index <.SPX> was down 14.47 points, or 1.19 percent, at 1,204.42. The Nasdaq Composite Index <.IXIC> was down 33.42 points, or 1.30 percent, at 2,546.04.

The benchmark S&P gained more than 5 percent during the four-day rally that ended on Wednesday on increasing hopes for a new stimulus plan from the Federal Reserve at its meeting in late September.

Ciena Corp , a communications equipment maker, jumped 20 percent to $14.71 after posting a profit for the first time in three years. Cisco Systems Inc , the network equipment maker, gained 1 percent to $15.82 and led the Dow.

The S&P retail index <.RLX> fell 1.2 percent as retailers reported August same-store sales that were slightly below expectations as Hurricane Irene drove business away from some stores. Target Corp fell 1.2 percent to $51.06 while Costco Wholesale Corp added 1.2 percent to $79.48.

U.S. construction spending fell unexpectedly in July as public outlays dropped to their lowest level since December 2006 and private spending also sagged, separate data showed.

Weekly jobless claims declined by 12,000 in the latest week, while nonfarm productivity was weaker than previously thought in the second quarter.

Eleven stocks fell for every four that rose on the New York Stock Exchange while on the Nasdaq one stock rose for every four that fell.

Volume was light, with about 7.49 billion shares traded on the New York Stock Exchange, the American Stock Exchange and Nasdaq, below last year's daily average of 8.47 billion.

(Editing by Leslie Adler)