Bernie Madoff is dead, and it is unlikely that even the people who were once closest to him will shed a tear. Not his wife, Ruth, whose life was destroyed when Madoff's Ponzi scheme was revealed in December 2008. Not his brother, Peter, Madoff's former chief compliance officer, who spent nearly a decade in prison after pleading guilty to a variety of charges. Not his niece Shana, Peter Madoff's daughter, who also worked in the compliance department. And certainly not his daughters-in-law or his grandchildren; both his sons died after he went to prison, one from suicide and the other from cancer.

By all appearances, Madoff loved his family deeply. He once held his son Mark's hand for the duration of a dinner because Mark was going through a difficult divorce, according to Town & Country. But that turned out to be just another fraud. He had to know that once his decadeslong Ponzi scheme came to light, it would be ruinous not just to his firm's clients but to his family. The word "sociopath" is overused these days, but it fits Madoff to a tee. He had neither empathy nor a conscience. That's the mentality required to rip off thousands of people who are depending on you.

Madoff's Ponzi scheme is what's called affinity fraud — "the pleasant-sounding term criminologists use when one member of a close-knit, trusting community exploits that trust to steal from others in the group," as Madoff biographer Diana Henriques put it. Madoff was Jewish, and his community were his fellow Jews.

Some, like Fred Wilpon, the former owner of the New York Mets, were wealthy. But many were not. Middle-class Jews gave him their money in the belief that their growing nest eggs would pay for their children's college tuition or their retirement. Madoff was adept at gaining people's trust. When they discovered that those gains were nothing more than an illusion, they were crushed. I once met a TSA guard at the Las Vegas airport. She was elderly, and after she looked at my driver's license, she told me how much she missed New York.

"What's keeping you from moving back?" I asked her flippantly. "Madoff," she said. "I don't have any money anymore."

In the months after Madoff's arrest, I was pretty unsympathetic to the plight of those who had lost money with Madoff. I thought they should have realized that it's implausible for a money manager to generate the kind of steady returns that Madoff did.

But then someone showed me a statement from Bernard L. Madoff Investment Securities LLC. I was astonished. It was extremely detailed, with a long list of securities that the client supposedly owned, along with monthly gains or losses. It must have been arduous for those helping Madoff commit the fraud to compile — and have it add up to a small gain each month. But if you were a relatively unsophisticated investor, could you truly be expected to even suspect that the statement was fraudulent? Unlikely.

In real dollars, the Madoff Ponzi scheme lost around $19 billion. According to the trustee for the Madoff estate, $14.4 billion has been recovered so far — an extraordinary accomplishment. But that does not suggest that Madoff's former clients are close to being made whole. You see, when you add in the fictitious gains that the victims thought they had, the amount comes to more than $64 billion. Even if the trustee, Irving Picard, were to recover the entire $19 billion, the victims would still be out $45 billion. Though that money never truly existed, it was very real in the minds of the victims who were counting on it.

What's more, many of the victims were sued by Picard, who tried to claw back money that Madoff had paid out to them. The theory — and it is very common in the aftermath of a Ponzi scheme — is that because that money was stolen, it didn't belong to the recipient. Most people wound up settling with the trustee — which sometimes meant selling a home or making other painful financial sacrifices. Many victims wound up embittered not just because of Madoff but because of the trustee as well.

Last summer, at the age of 82, Madoff petitioned the court for a compassionate release, something the Federal Bureau of Prisons has begun to grant elderly prisoners who are sick. In 2009, he was sentenced to 150 years in prison. But now, he said, he was terminally ill with kidney failure and was likely to die within 18 months.

Bernie Ebbers, the former chief executive officer of WorldCom, had been granted a compassionate release six months earlier. Madoff's brother, Peter, had spent a portion of his sentence in home confinement. Donald Trump's campaign chairman Paul Manafort was released (before his pardon). Michael Cohen, Trump's former lawyer, has been allowed to finish his sentence at home.

But not Madoff. "When I sentenced Mr. Madoff in 2009, it was fully my intent that he live out the rest of his life in prison," U.S. Circuit Judge Denny Chin said. "Nothing has happened in the 11 years since to change my thinking." Less than a year later, he died alone — no family, no friends, no former colleagues. Which, given all the harm he caused to so many people, is exactly what he deserved.

Joe Nocera is a Bloomberg Opinion columnist covering business. He has written business columns for Esquire, GQ and the New York Times, and is the former editorial director of Fortune. His latest project is the Bloomberg-Wondery podcast "The Shrink Next Door."