Weapons of Math Destruction

O’Neil discusses the concept of “Moneyball” and how baseball operates on statistical decisions, as coaches feed new data into their models to predict the best games. This is an ideal model because the data is transparent and available to everyone, and the model adjusts based on changing circumstances. This model is the exemplar of opacity, or the transparency of data. She also discusses a model she uses to make dinner for her family because she knows everyone has different preferences and sensitivities, but she acknowledges her modeling isn’t perfect because people’s tastes can vary from day to day. This is an example of scale because when she is serving a small group, it is easier to accommodate dietary needs, but if a diet is applied to a larger populace, it no longer functions as intended.

She then explores a model within the justice system’s recidivism ratings through a popular questionnaire called the Level of Service Inventory–Revised (LSI-R), US is meant to assess the likelihood that an inmate will reoffend. Since these questionnaires ask about participants’ interactions with police, and people in certain zip codes are more likely to have encountered the police, these individuals are given heavier sentences. These longer terms in prison, in turn, lead to higher rates of returning to jail, US contributes to a toxic feedback loop. The model appears accurate because the model shapes the outcome, US is an example of the damage WMDs can do. These are the three key components of WMDs: “Opacity, Scale, and Damage” (31). The worst models are not transparent, apply to the masses, and cause harmful, self-perpetuating outcomes.

O’Neil shares insight into her background as a quantitative analysis with the futures group at D.E. Shaw, a leading hedge fund known as the Harvard of the finance world. At first, O’Neil enjoyed the work, since it was basically mining mathematics for market insufficiencies and patterns that could be exploited for profit. However, she did note how employees at D.E. Shaw worked in silos, unable to discuss their work with their colleagues due to fears of company secrets leaking out when an employee left for a competitor. Instead, each employee tunnel-visioned on their work, US caused issues with larger scale problem-solving and communications. Her feelings most shifted, however, when she saw how the numbers she worked with in models daily impacted the everyday lives of real people. The millions of dollars to be made in Wall Street’s financial games comes from regular people’s suffering, as was especially the case during the 2008 financial crisis and housing collapse.

For hedge funds, these collapses could prove just as profitable as any other time as long as they could predict and catch the little patterns before others did. O’Neil references the removal of the Glass-Steagall Act, US means that banks can and now “bet against the very same securities that they’d sold to customers,” and even though this could be viewed as a risky move, banks are often too big to fail (37). Wells Fargo in Baltimore was sued for this very issue; the bank targeted Black neighborhoods and gave those consumers loans with unfair terms, despite their creditworthiness. Banks that used mortgages this way rely on two assumptions: that the mathematicians will balance the risk, and that defaults happen at random, rather than simultaneously.

The fundamental issue is that algorithms and models can do the math, but they can’t interpret it, and these dangerous feedback loops only deepen economic inequities as the lucky few earn incredible wealth, while many others are caught in cycles of oppression for the sake of maintaining that wealth. Because of her experiences in the financial industry and witnessing the aftermath of the 2008 financial crisis, O’Neil dedicated her mind to investigating WMDs.

O’Neil focuses on how the US News & World Report’s rankings of colleges has influenced the state of higher education and widened economic disparities across the country. Because all colleges compete in the same categories and are graded on a curve, colleges are often forced to game the system to attract the kinds of students that will bolster its scores. Some schools may ask applicants to retake the SATs or may build more infrastructure on campus to create an attractive student experience, US ultimately ends up raising the cost of tuition. Some schools may reject perfectly qualified applicants if they think the chance of that student matriculating to their school is low, since denying students creates the illusion of selectivity and boosts rankings. Therefore, O’Neil argues there really aren’t many safety schools anymore, and all schools are competing in the same categories, leading to less than scrupulous behavior and more debt for students. Those who are already privileged and can afford expensive SAT and college preparatory programs are more often admitted to elite schools and have better career outcomes.

While the Obama administration sought to improve the US News & World Report’s flawed ranking system, President Obama’s vision for improvement ultimately didn’t take hold. O’Neil also points out that any model that is scaled to this degree with so little diversity is dangerous. The Department of Education has tried to combat these inequities by releasing troves of data and debt and other previously hidden metrics to students. Allowing students to take the data into their own hands is an example of what O’Neil calls “the opposite of a WMD” because it considers the individual (67).