It's no secret that falling behind on student loan payments can squash a borrower's hopes of building savings, buying a home or even finding work. Now, thousands of retirees are learning that defaulting on student debt can threaten something that used to be untouchable: their Social Security benefits.

According to government data compiled by the Treasury Department at the request of SmartMoney.com, the federal government is withholding money from a rapidly growing number of Social Security recipients who have fallen behind on federal student loans. From January through Aug. 6, the government reduced the size of 115,000 retirees' Social Security checks on those grounds.

That's nearly double the pace of the department's enforcement in 2011; it's up from around 60,000 cases in all of 2007 and just six cases in 2000.

The amount that the government withholds varies widely, though it runs up to 15 percent. Assuming the average monthly Social Security benefit for a retired worker of $1,234, that could mean a monthly haircut of almost $190.

Many of these retirees aren't even in hock for their own educations. Consumer advocates say that in the majority of the cases they have seen, the borrowers went into debt to help defray education costs for their children.

Most consumer advocates and attorneys who work with seniors in this predicament said their clients were unwilling to speak on the record, because of shame or fear.

For example, Deanne Loonin, a staff attorney at the National Consumer Law Center in Boston, said she's been working with an 83-year-old veteran whose Social Security benefits have been reduced for the past five years. The client fell behind on a federal loan that he signed up for in the '90s to help with his son's tuition costs; Loonin said the government's cuts have left the client without enough cash to pay for medications.

About 2.2 million student-loan debtors were 60 and older in the first quarter of 2012, according to the Federal Reserve Bank of New York.

"It's really a unique problem we haven't had to face before, and it's only going to grow," said Robert Applebaum, founder of Student Debt Crisis, a nonprofit advocacy group in Staten Island, N.Y.

The threat of Social Security cuts adds to the overall financial woes faced by the aging baby boomer generation.

Almost 45 percent of people ages 48 to 64 won't save enough money to cover basic needs and uninsured health-care costs in retirement, according to the Employee Benefit Research Institute.

That same generation has been slammed by the soaring cost of college, whether for their kids or themselves.

No wiping away of debt

Unlike other consumer debts, student loans typically can't be wiped out in bankruptcy. And changes in the law over the past couple of decades have given Uncle Sam more power to pursue defaulters, said William Brewer, president of the National Association of Consumer Bankruptcy Attorneys.

The Debt Collection Improvement Act of 1996 empowered the federal government to offset Social Security payments of defaulted student-loan borrowers. An earlier law, the Higher Education Technical Amendments Act, essentially removed any time limits on the government's ability to collect from the defaulters. The Supreme Court upheld both provisions in a 2005 ruling.

The government's withholding power also extends to Social Security disability benefits. Tammy Brown of Redding, Calif., said the government has been taking $179 out of her Social Security disability check each month for the past five years. Brown, 52, became disabled in 1986 after being involved in a car accident. Unable to work, she fell behind on her student loan payments. She said the Social Security check is now too small to cover her food and medical bills, so she quit taking prescription pain pills.

"It's kind of hard to live on this amount of money," she said.

The Department of Education, which provides federal student loans to borrowers, says it tries to work out payment plans with people who fall behind.

For its part, the Treasury Department said it reaches out to borrowers twice to set up a payment plan or otherwise resolve their debt before offsetting money from their Social Security check. And the Treasury won't withhold money from monthly checks that total $750 or less, said Ronda Kent, deputy assistant commissioner for debt management services at the Treasury Department's Financial Management Service.