What Happened
A study by Obermeyer and colleagues published in Science found that a commercial algorithm used to select patients for high-risk care-management programs was racially biased. Because it used healthcare spending as a proxy for health need, and less is historically spent on Black patients, Black patients had to be considerably sicker than white patients to receive the same risk score. Algorithms of this type were applied to the care of roughly 200 million Americans annually.
Impact
Correcting the bias would have raised the share of Black patients flagged for extra care from 17.7% to 46.5%, meaning huge numbers of sick patients were passed over. The study triggered a New York regulatory inquiry and became foundational evidence that proxy-label choices can encode systemic bias at population scale.
How to Prevent This
- Validate that the prediction target (label) actually measures the outcome you care about, not a biased proxy like cost
- Stratify model performance by race and other protected groups before deployment
- Give external researchers audited access to production algorithms affecting care decisions
- Recalibrate or retrain when subgroup miscalibration is detected, and re-verify after fixes
- Require bias impact assessments for any algorithm allocating scarce medical resources