What Happened
The Markup found that New York City's Microsoft-powered MyCity chatbot, launched to help small businesses navigate regulations, confidently gave illegal advice: that employers could take workers' tips, landlords could refuse Section 8 vouchers, businesses could refuse cash, and bosses could fire employees for reporting harassment. All of these contradict city, state, or federal law. Mayor Eric Adams acknowledged the errors but kept the bot online.
Impact
Business owners risked legal liability by following official government guidance that was flat wrong, and the city faced sustained criticism for leaving a known-defective chatbot publicly deployed.
How to Prevent This
- Have legal experts validate chatbot answers against current statutes before launch
- Restrict government chatbots to retrieval of verified official guidance with linked sources
- Display prominent disclaimers and human referral paths for legal and compliance questions
- Continuously audit live answers against a test suite of known-law questions
- Define takedown criteria in advance and suspend the service when error rates exceed them