What Happened
Amazon's AI recruiting tool was trained on 10 years of resumes (mostly from men) and learned to penalize resumes containing the word "women's" (as in "women's chess club") and downgrade graduates of all-women's colleges.
Impact
Tool scrapped after years of development. Highlighted how AI perpetuates historical bias. Led to industry-wide scrutiny of hiring algorithms.
Cost: Years of R&D costs written off, reputation damage
How to Prevent This
- Audit training data for demographic representation
- Test for bias across protected characteristics
- Diverse teams building and testing AI systems
- Regular fairness audits before deployment