Weapons Of Math Destruction

Statistical models are used everywhere in our day and age. They can be used to do great good, like estimating where child abuse is likely happening, but also for great evil, like influencing elections in foreign countries.

WMDs are statistical models with a flaw. They’re missing the feedback loop, a critical part of a models lifecycle. This results in models that can’t be evaluated on real life performance. They’re “flying blind”, unaware of the damage they cause and unable to change.

Cathy O’Neil says that a WMD always meets three criteria.

1. Opacity
2. Scale
3. Damage

That is, 1) it’s impossible for outsiders to understand how the model draws conclusions (e.g. what input data is used), 2) it does affect a large number of people, and 3) it has a negative impact on at least one person.

She then gives a multitude of examples from many parts of life. Education, law enforcement, advertisement, employment, and insurance. Nothing is safe from a WMD.

Cathy concludes that people creating statistical models should take an oath similar to the oath taken by doctors.