The U.S. wealth management industry is shrinking, losing legions of advisers, just as demand for their services is on the rise.

The ranks of brokers and investment advisers working at brokerages, banks, insurers and independent firms fell by nearly 14,000, or 4.1 percent, last year, according to the latest Cerulli Associates report that tracks the movement of people and assets among U.S. firms.

Meanwhile the four largest brokerages -- Morgan Stanley Smith Barney, Bank of America's Merrill Lynch, Wells Fargo Advisors and UBS Wealth Management Americas -- are losing advisers and clients to independent broker-dealers and investment advisers, according to Cerulli.

That means Wall Street's big firms, which for years have been paying up to recruit star brokers from each other, are likely in for bigger bills, Cerulli analyst Bing Waldert said.

Industry-wide, the adviser population hasn't been growing for some time, and its shrinkage accelerated in 2010, said Waldert, co-author of the widely followed report.

The decline in headcount, following years of stagnation, partly reflects an adviser population nearing retirement age. It also reveals the shake-out of laggard brokers and advisers whose livelihoods have been hurt since the financial crisis.

Banks and other parts of the wealth management industry employed about 320,000 advisers as of the end of last year, little changed for seven years. By 2015, total headcount may fall by another 8,000 people to about 312,000 advisers.

It raises questions about how the industry is going to take advantage of all the Baby Boomers heading into retirement, if firms don't have new talent coming in, he said.

The big national brokerages, wirehouses in industry parlance, are losing advisers who seek the higher payouts and greater control over their practices that comes with being an independent broker, Waldert said.

During the past four years, the ranks of registered investment advisers (RIAs) grew nearly 7 percent a year while brokers employed by the big national firms fell 3.7 percent.

The breakaway trend, though, is less dramatic than commonly portrayed. Movement is down dramatically since record numbers of brokers jumped to rivals after the 2008 financial crisis.

Independent brokerages overall saw their headcount fall by less than 1 percent since 2007. A shakeout among struggling smaller firms, and a slowdown in recruiting wins, offset gains by top players like LPL Investment Holdings Inc.

There is a very real, secular trend toward independence, Waldert said, with self-employed advisers gaining about one to two percentage points in market share each year at the expense of national and regional brokerages.

Last year, self-employed advisers represented 41 percent of the industry's headcount, up from 37 percent in 2004. The RIA channel has grown by 6.7 percent over the past three years, as growing numbers of investors seek advisers who are not paid based on trades and product sales.

Assets managed by independent brokers and advisers grew to nearly 35 percent from 30 percent in just four years.

Wall Street's banks, to be sure, have been culling hundreds of laggard performers to reduce costs and boost profits.

Regional brokerages, a group that includes Raymond James Financial and Edward Jones, saw their adviser ranks fall by nearly 4,000, or 10 percent, over the same period.

And the shrinkage will continue for the Big 4 firms. Cerulli sees their market share dropping 7 percentage points by 2013, leaving them with 35 percent of industry-wide client assets. RIAs by the same token will boost their share by 2.5 points to reach 14 percent.

Independent broker-dealers, lately a hotbed of acquisition activity, are expected to boost their market share by 2 points to 18 percent of assets over the next few years, Cerulli said.

Executives at the big brokerages play down the breakaway trend as overblown. Indeed, they command twice as many assets as independent brokers, the runner-up group.

Still, the big guys face a difficult task of increasing their sales forces and retaining their market position.

The industry has to figure out how to productively bring new talent into the industry, Waldert said. People don't come to the wirehouses from other channels. It's a one-way street.