Americans agonize each spring over their annual federal income tax returns. In terms of how much tax they pay, however, they should worry more about the payroll taxes their employers withhold from every paycheck. After all, three-quarters of those who pay one or both taxes shell out more for payroll taxes than for income tax.

The Tax Policy Center estimates that 44 percent of households will pay no federal income tax this year (down from the 47 percent famously cited by Mitt Romney in 2012). But 60 percent of those households have members who work and will thus pay Social Security and Medicare payroll taxes. As the light blue block shows, that’s about one-quarter of all households.

Nearly half of all households will pay both income and payroll taxes (the dark yellow and dark blue blocks). Almost three-fourths of them will pay more in payroll taxes than in income taxes. Just 7 percent of households (the light yellow block) will owe income tax but no payroll tax because they don’t have any wage income.

Source: Tax Policy Center, Distribution of Federal Payroll and Income Taxes by Expanded Cash Income Percentile, 2016

Overall, almost two-thirds of households will pay more payroll tax than income tax (the two blue blocks), while only one in five will pay more income tax (the two yellow blocks).

And what about the remaining 18 percent who pay neither tax? More than half are retirees living on Social Security and too little other income to owe income tax. The rest have no jobs and not much income. That isn’t a club many would want to join.