Determining whether or not you have to pay taxes on your Social Security benefits really comes down to two factors: 1) your tax filing status, and 2) the amount of your "combined" income. Generally speaking though, you will not have to pay any taxes on your Social Security benefits at all, if that is your only source of income.