Security team helps victims of ransomware attack, including Crystal Valley co-op in Minnesota

In a year rife with ransomware attacks, when cybercriminals have held the data of police departments, grocery and pharmacy chains, hospitals, pipelines and water treatment plants hostage with computer code, it was a win, rare in the scale of its success.

For months, a team of security experts raced to help victims of a high-profile ransomware group quietly recover their data without paying their digital assailants a dime.

It started in late summer, after the cybercriminals behind the Colonial Pipeline ransomware attack, known as DarkSide, emerged under a new name, BlackMatter. Soon after, the cybercriminals made a glaring mistake that most likely cost them tens, if not hundreds, of millions of dollars.

Ransomware criminals encrypt a victim's data and demand a ransom payment, sometimes millions of dollars, to return access. But when BlackMatter committed a critical error in an update to its code, researchers at Emsisoft, a cybersecurity firm in New Zealand, realized they could exploit the error, decrypt files and return access to the data's rightful owners.

Emsisoft hustled to track down dozens of victims in the United States, Britain and Europe so it could help them secretly unlock their data. In the process, the firm kept millions of dollars in cryptocurrency out of the cybercriminals' coffers.

It was a short-lived victory in the cat-and-mouse game of ransomware, which is expected to cost organizations $20 billion in losses this year, according to a report from research firm Cybersecurity Ventures. It was so unusual, even the victims whose data was saved by the effort could not believe it. Many thought Emsisoft was running a scam.

Emsisoft officials described their operation, which has not been reported before, in a series of interviews with the New York Times.

To assuage victims' concerns, Emsisoft researchers asked their contacts at cybersecurity companies and government agencies around the world to vouch for them.

While Emsisoft would not identify the victims, it said they had included key manufacturers, transportation companies and food suppliers across continental Europe, Britain and the United States.

The timeline of Emsisoft's effort overlaps with BlackMatter's ransomware assaults last month on two U.S. agriculture organizations: NEW Cooperative, an Iowa grain cooperative, and Crystal Valley, a Minnesota farming supply cooperative. Both cooperatives recovered quickly, suggesting that Emsisoft might have helped. Neither company returned requests for comment.

Eric Goldstein, executive assistant director for cybersecurity at the federal Cybersecurity and Infrastructure Security Agency, called the effort a model for public and private collaboration. The agency is trying to develop a comprehensive "whole of nation" plan to address cyberthreats, particularly for "critical infrastructure," most of which is owned by the private sector.

CISA recently created the Joint Cyber Defense Collaborative, which teams government agencies with tech firms like Microsoft and Amazon, telecommunication firms like AT&T and Verizon, and cybersecurity firms like CrowdStrike and Palo Alto Networks to address threats like ransomware.

That effort followed several smaller victories against REvil over the summer. The group, which is responsible for thousands of ransomware attacks, found itself in the government's crosshairs after it pulled off a high-profile attack on JBS, one of the world's biggest meatpacking operators, and Kaseya, a Miami software company. The group used Kaseya's high-level access to its customers to hold hundreds of them hostage over this past Fourth of July holiday.

A week later, REvil's websites went dark, leading to speculation that governments may have played a role. A week after that, Kaseya announced that a mysterious "third party" had given it the key to unlock its customers' encrypted data. In fact, the FBI later confirmed that it had secured a key but delayed giving it to Kaseya's customers while it coordinated with other agencies to take down the group. But before it could act, REvil went offline on its own.

REvil reappeared in September, before disappearing again last week.

Recent history suggests REvil's operators could just reemerge under a new name. As long as ransomware groups enjoy immunity in Russia and other nations, ransomware continues to plague American companies and organizations. The latest to fall victim appears to be the police in Hagerstown, Maryland. On Friday, the same cybercriminals who hijacked and then leaked sensitive data from the Washington, D.C., Police Department in April, claimed to have breached the Hagerstown police website and stolen the login credentials. Contacted late Friday, Hagerstown police said they did not believe that employee data was stolen but were closely monitoring the situation and had changed passwords and taken other mitigation steps.

American cybersecurity officials concede that beyond a few brief triumphs, there has been no material shift in Russian cyberattacks since President Joe Biden's first summit with Russia's president, Vladimir Putin, in June. Biden warned Putin that attacks on America's 16 critical infrastructure sectors — like the food suppliers hit last month — could warrant retaliation.

The noise around the NEW Cooperative attack created additional challenges for Emsisoft, the company said. Emsisoft had been finding BlackMatter victims through posts to a Google-owned platform, VirusTotal, which is a kind of search engine for malware.

Those posts helped link Emsisoft's teams to the chat platform that BlackMatter used to negotiate ransom payouts with its victims. Emsisoft monitored the chats to see if cybercriminals or victims dropped the name of their organization, then used that information to contact the victims.

But after NEW Cooperative's attack made headlines, unexpected visitors started leaving insults in chat rooms where BlackMatter negotiated payments. When BlackMatter threatened to leak NEW Cooperative's data online for violating its "data recovery guidelines," someone replied with an unsavory insult directed at a BlackMatter criminal's mother.

A representative for NEW Cooperative made clear in the chat that the comment had come not from them but from "random people from the internet." The exchange prompted BlackMatter to shut down access to its online chats and start vetting anyone who entered. In the process, Emsisoft lost a key way to reach the victims.

Emsisoft knew it could not publish its secret ability without tipping off BlackMatter. But the company was still able to reach several BlackMatter victims whose data had been posted online. (To add pressure, ransomware groups now post a victim's information online when it refuses to pay.)

"The reason ransomware operators have gotten away with so much crime is that, until recently, there's been far too little cooperation and communication all around," said Brett Callow, a threat analyst at Emsisoft. "This shows that private/public-sector cooperation can put a significant dent in their profits."

Emsisoft knew it was running out of time. Inevitably, BlackMatter would start to wonder why so many victims stopped paying their ransoms or why many did not even bother to respond.

Finally, last month, BlackMatter caught the mistake. It was back to the drawing board for researchers at Emsisoft and other companies.