The election in North Rhine-Westphalia, a western German state with a bigger population than the Netherlands and an economy the size of Turkey, was held 18 months before a national election in which Merkel is expected to fight for a third term.
She remains popular in Germany for her steady handling of the euro zone debt crisis, but the sheer scale of her party's defeat leaves her vulnerable at a time when a backlash against her insistence on fiscal discipline is building across Europe.
According to first projections, the center-left Social Democrats (SPD) won 38.8 percent of the vote and will have enough to form a stable majority with the Greens, who scored 12.2 percent.
The two left-leaning parties had run a fragile minority government for the past two years under popular SPD leader Hannelore Kraft, whose decisive victory on Sunday could propel her to national prominence.
Merkel's Christian Democrats (CDU) saw its support plunge to just 25.8 percent, down from nearly 35 percent in 2010, and the worst result in the state since World War II.
This is not a good evening for Merkel, said Gero Neugebauer, a political scientist at Berlin's Free University.
The SPD is strengthened by this election, which will stir things up in Berlin.
The blow comes only two days before France's new president, Socialist Francois Hollande, is due to visit Berlin and press Merkel for a shift away from austerity and more emphasis on growth-oriented measures in Europe.
Other big countries like Italy also want Merkel to take a more balanced approach to the debt crisis and an election in Greece last week showed massive public resistance to tough austerity.
Hollande's victory, coupled with the NRW result, is bound to give the SPD, which still trails Merkel in national opinion polls, new momentum before the federal vote in the autumn of 2013.
The chancellor needs the support of her rivals to pass a new fiscal compact that is meant to anchor budget discipline across the EU. The SPD is already pressing her to delay a parliamentary vote on the pact, keen for her to commit to new growth measures beforehand.
Many in her party will blame the result on regional leader Norbert Roettgen, Merkel's environment minister in Berlin, who bungled his campaign early on by refusing to commit to staying in the state in the event of a loss.
Roettgen ran on a platform of budget consolidation in a state that, with 180 billion euros in debt, is Germany's most indebted. Kraft, on the other hand, advocated a go-slowly approach to debt reduction, emphasizing the need to invest in cities, education and childcare.
In that sense, the result will be seen by some as a double defeat for Merkel. Voters in NRW not only rejected her party but also the austerity measures that she has forced on struggling southern states like Greece, Spain and Portugal.
The Free Democrats, a pro-business party that rules in coalition with Merkel's conservatives at the federal level, scored 8.6 percent to make it back into the state assembly. The party hailed the result as proof of a comeback after a collapse in support over the last three years.
The upstart Pirates, a new party that campaigns for Internet freedom, continued its strong run at the regional level, making it into its fourth straight state parliament, winning 7.6 percent of the vote.
NRW, a diverse state with struggling cities in the rust-belt Ruhr region and home to one-third of Germany's blue-chip companies, has a history of influencing national politics.
Seven years ago, a humiliating loss for then-Chancellor Gerhard Schroeder's SPD in the state prompted him to call early elections, which he subsequently lost to Merkel.