Will retirement income curb Social Security benefits? If you're single and earn more than $25,000 or married with household income over $32,000, yes. In that case, up to 85% of your Social Security benefits are taxable a huge hit. So it generally does not pay to begin collecting Social Security until your income from both wages and, importantly, IRA and 401(k) withdrawals falls below those thresholds. Passive income from things like stock dividends and bond interest do not count in this calculation. Meanwhile, by deferring Social Security, you'll get a higher monthly benefit after age 70, when you should start collecting no matter what because no additional benefits will accrue.