The income gap between America’s rich and poor has widened during the economic recovery -- despite rising employment and a growing economy, the Federal Reserve said Thursday.

While average income before taxes for the wealthiest 10 percent of U.S. families rose 10 percent from 2010 to 2013, inflation-adjusted incomes for the poorest 40 percent of families actually declined, according to the Fed’s Survey of Consumer Finances.

The average income for all Americans rose 4 percent from 2010, but the median income—the amount in the middle of all Americans’ incomes and not skewed by ultra-high salaries — fell 5 percent, the report said.

Only a sliver of the wealthiest Americans have seen widespread income gains since the recession. 

The income of the richest 3 percent of families increased from a 27.7 percent share of all U.S. income in 2010 to a 30.5 percent share in 2013, nearly the 31.4 percent share that group held in 2007.

“It’s fair to say the economic recovery has helped people who have very high-paid positions more than it has helped people who have low-paying jobs,” Gary Burtless, senior fellow and economist at the Brookings Institute, said. “But people at the very top of the income distribution took bigger hits to their income [during the financial crisis] than people near the bottom.”

So while the recovery has been slow, it's normal for the wealthiest to recover their financial footing first, and middle- and lower-income families should see gains next, Burtless said.