What Happened
After moving to replace its human helpline staff, who had voted to unionize, NEDA deployed the Tessa chatbot to support people with eating disorders. Users reported that Tessa recommended calorie counting, daily deficits of up to 1,000 calories, weekly weight loss targets, and buying skin calipers to track body fat — exactly the behaviors that fuel eating disorders. NEDA disabled the bot after the reports went viral.
Impact
The chatbot was taken offline indefinitely and NEDA faced intense criticism for replacing trained human counselors with an unsafe automated system. The episode became a canonical example of deploying AI in a clinical-adjacent role without adequate safety validation.
How to Prevent This
- Clinically validate any health-adjacent chatbot with domain experts before and after every model or vendor change
- Pin high-risk topics (weight loss, calories) to reviewed, static responses instead of generative output
- Keep a human-in-the-loop fallback for vulnerable populations rather than fully replacing trained staff
- Continuously monitor production transcripts for advice that contradicts clinical guidelines
- Treat vendor-added AI capabilities as new deployments requiring fresh safety review