Only 14 percent of U.S. residents pay neither income tax nor payroll taxes, and most of them are elderly or make less than $20,000 in annual income, according to a new report from the Tax Policy Center.

The number of people who pay both federal income tax and payroll tax increased from 53.1 percent in 2011 to 56.7 percent in 2012. This is likely due to an improving economy and the expiration of temporary recession tax cuts, according to the Tax Policy Center report.

28.9 percent don’t pay federal income tax but do pay payroll tax. Even the 14 percent who pay neither payroll tax nor income tax do pay Social Security, state and local taxes, or some other levy, said the report.

And the remaining 1 percent mostly benefit from the tax code’s many exclusions, deductions, exemptions and credits that wipe out the income tax they would otherwise owe, according to the report.

who pays tax
Only 14 percent of U.S. residents pay neither income nor payroll taxes, and most of them are elderly or poor. IBTimes/Lisa Mahapatra

Correction: An earlier version of this infographic extrapolated population numbers from the percentages calculated by the Urban-Brookings Tax Policy Center Model. However, these percentages refer to tax filing units, not the U.S. population. The infographic was edited for accuracy.