A widely used algorithm that predicts which patients will benefit from extra medical care dramatically underestimates the health needs of the sickest black patients, amplifying racial disparities in medicine, researchers found.
The problem was caught in an algorithm sold by a leading health services company, called Optum, to guide health care decisionmaking for millions of people. But the same issue almost certainly exists in other tools used by other U.S. private companies, nonprofit health systems and government agencies to manage the care of about 200 million people each year, the scientists reported in the journal Science.
Correcting the bias would more than double the number of black patients flagged as at risk of complicated medical needs and researchers are already working with Optum on a fix. When the company replicated the analysis on a national data set of 3.7 million patients, they found that black patients who were ranked as equally in need of extra care as white patients were much sicker: They collectively suffered from 48,772 additional chronic diseases.
“I’m hopeful that this causes the entire industry to say, ‘Oh, my, we’ve got to fix this,’ ” said Sendhil Mullainathan, a professor at the University of Chicago Booth School of Business, who oversaw the work.
Machines increasingly make decisions that affect human life, and big organizations — particularly in health care — are trying to leverage massive data sets to improve how they operate. They utilize data that may not appear to be racist or biased, but may have been heavily influenced by longstanding social, cultural and institutional biases.
There is a long history of black patients facing barriers to accessing care and receiving less effective health care.
Mullainathan and his collaborators discovered that the algorithm they studied, which was designed to help target patients who would have the greatest future health care needs, was actually predicting how likely people were to use a lot of health care and rack up high costs in the future.
Since black patients incurred about $1,800 less in medical costs per year than white patients with the same number of chronic conditions, the algorithm scored white patients as equally at risk of future health problems as black patients who had many more diseases. The algorithm would then deepen that disparity by flagging healthier white patients as in need of more intensive care management.
“Predictive algorithms that power these tools should be continually reviewed and refined,” Optum spokesman Tyler Mason said.
Ruha Benjamin, an associate professor of African American studies at Princeton University, said, “I am struck by how many people still think that racism always has to be intentional and fueled by malice. They don’t want to admit the racist effects of technology unless they can pinpoint the bigoted boogeyman behind the screen.”
Algorithms are notoriously opaque because they are proprietary products. In the new study, researchers had an unusual amount of access to the data that went into the algorithm. The researchers also found a relatively straightforward way to fix the problem. Instead of just predicting which patients would incur the highest costs and use the most health care in the future, they tweaked the algorithm to make predictions about their future health conditions.