Weapons of Math Destruction

“[R]acism is the most slovenly of predictive models. It is powered by haphazard data gathering and spurious correlations, reinforced by institutional inequities, and polluted by confirmation bias. In this way, oddly enough, racism operates like many of the WMDs I’ll be describing in this book.”

The racial implications of WMDs echo throughout the book and dramatically alter society so that it is skewed in favor of the privileged. While many of these WMDs seem complex in that they are hidden from public view and nefarious, they are quite simple in that they are human stereotyping in machine form.

“It wasn’t found money, like nuggets from a mine or coins from a sunken Spanish galleon. This wealth was coming out of people’s pockets. For hedge funds, the smuggest of the players on Wall Street, this was ‘dumb money.’”

The extreme wealth afforded to the elite few isn’t harmless profit. It is money skimmed from those who need it the most, and the elite maintain their positions by finding ways to further exploit the populations who keep them wealthy.

“The math could multiply the horseshit, but it could not decipher it.”

O’Neil’s crude language here may catch readers off-guard, but essentially functions to draw attention to the point: Math can do math, like multiplying, but it doesn’t interpret or analyze its calculations in any meaningful way.

“A formula, whether it’s a diet or tax code, might be perfectly innocuous in theory. But if it grows to become a national or global standard, it creates its own distorted and dystopian economy.”

The scale of a WMD and the assumption of its fairness due to the clinical nature of its calculations leads to a society grossly divided by class. Like many dystopias, it applies the rules or beliefs of the few onto the many without consideration for individual circumstances or needs.

“The victims, of course, are the vast majority of Americans, the poor and middle class families who don’t have thousands of dollars to spend on courses and consultants. They miss out on precious insider knowledge. The result is an education system that favors the privileged. It tilts against needy students, locking out the great majority of them—and pushing them down a path toward poverty. It deepens the social divide.”

Education is often touted as a gateway to freedom and opportunity, but if education leads to better outcomes, the exclusion of access to quality education can lead to the converse: being trapped and without options. The more trapped people are, the fewer options they have, the more desperate they become.

“[For-profit universities] find inequality and feast on it. The result is that they perpetuate our existing social stratification, with all of its injustices. The greatest divide is between the winners in our system, like our venture capitalist, and the people his models prey upon.”

Using the promise of education leading to better opportunities in the future, for-profit colleges also exploit the downtrodden by preying upon their deepest desires for security and upward mobility and, in turn, locking them into cycles of poverty with massive debt.

“An advertising program might start out with the usual demographic and geographic details. But over the course of weeks and months it begins to learn the patterns of the people it’s targeting and to make predictions about their next moves. It gets to know them. And if the program is predatory, it gauges their weaknesses and vulnerabilities and pursues the most efficient path to exploit them.”

While advertisers may seem to only rely on basic assumptions about the populations they target, it’s actually a lot more complicated. The most prolific advertisers track individuals to learn about their behavior, wants, needs, and fears to predict what we might purchase and create a market for it. On its face, this is the heart of advertising, capitalizing on a market, but beneath that lurks something much more nefarious: the control that the consumer’s lack of privacy affords them.

“So even if a model is color blind, the result of it is anything but. In our largely segregated cities, geography is a highly effective proxy for race.”

The mathematics may seem unbiased, but the feedback that is fed into the formulas is recursive without meaningful analysis, effectively perpetuating and expanding racism by starting with a formula based on location.

“Our livelihoods increasingly depend on our ability to make our case to machines.”

Being reliant on technology, though efficient, prevents individuals from changing their circumstances. Machines and algorithms don’t care about an individual’s life. These machines are designed not to care about anything other than following the formulas they were created to execute. As such, appealing to technology for fairness and understanding is impossible and leads many people not to bother questioning it in the first place.

“Under the inefficient status quo, workers had not only predictable hours but also a certain amount of downtime. You could argue that they benefited from inefficiency: some were able to read on the job, even study. Now, with software choreographing the work, every minute should be busy.”

The so-called inefficiency here leads to more than just the issue of downtime at work but also would free workers to elevate themselves and give them chances at better jobs and lives by allowing them time to work towards self-improvement beyond the company’s bottom line. In the long-term, this is dangerous to companies who rely on a workforce that has no chance at upward mobility and no bargaining power.

“People are hungry for work, which is why so many of them cling to jobs that pay barely eight dollars per hour. This oversupply, along with the scarcity of effective unions, leaves workers with practically no bargaining power. This means the big retailers and restaurants can twist the workers’ lives to ever-more-absurd schedules without suffering from excessive churn. They make more money while their workers’ lives grow hellish. And because these optimization programs are everywhere, the workers know all too well that changing jobs isn’t likely to improve their lot. Taken together, these dynamics provide corporations with something close to a captive workforce.”

Despite poor working conditions and few benefits, workers are hesitant to leave because they need work to survive and know that the low standard is the same across the board for the most part. By wearing people down to the point that they’re too exhausted and hopeless to fight for change, corporations are able to maximize profit at the expense of its workers’ well-being, an expense which costs them nothing.

“As e-scores pollute the sphere of finance, opportunities dim for the have-nots. In fact, compared to the slew of WMDs running amok, the prejudiced loan officer of yesteryear doesn’t look all that bad. At the very least, a borrower could attempt to read his eyes and appeal to his humanity.”

While it may seem like hyperbole to argue that the biased loan officer who probably engages in nepotism is a stronger alternative to the WMDs that govern the world of e-scores, O’Neil is using this exaggeration to make a point: When the humanity is taken out of decisions that affect humanity and only the (often biased) calculations remain, individuals who once had a low chance at success now have almost no chance at success. Technology doesn’t take an individual’s perspective into consideration because that’s not what it’s designed to do.

“Well, like other WMDs, automatic systems can plow through credit scores with great efficiency and at enormous scale. But I would argue that the chief reason has to do with profits. If an insurer has a system that can pull in an extra $1,552 a year from a driver with a clean record, why change it?”

The key point here is that the primary objective isn’t speed, so much as it is speed that leads to profits. By disregarding individual circumstances, blanket assumptions are applied, and more profits are reaped across the board by people who can’t dispute them. O’Neil uses this rhetorical question to emphasize how companies have no incentive to change unless it hits them where it hurts: the wallet.

“These automatic programs will increasingly determine how we are treated by the other machines, the ones that choose the ads we see, set prices for us, line us up for a dermatologist appointment, or map our routes. They will be highly efficient, seemingly arbitrary, and utterly unaccountable. No one will understand their logic or be able to explain it. If we don’t wrest back a measure of control, these future WMDs will feel mysterious and powerful. They’ll have their way with us, and we’ll barely know it’s happening.”

O’Neil paints an image of a dystopian, inhuman future governed by technology while the average citizen is unaware how much their life is being affected. However, the tone here, while bleak and even desperate, carries with it a seed of hope with the underlying message that it’s not too late to change the narrative and steer our society and its technology in a more positive direction.

“While Facebook may feel like a modern town square, the company determines, according to its own interests, what we see and learn on its social networks.”

O’Neil uses a simile to compare Facebook to a platform for a local community. This comparison calls attention to how nefarious this comfortable feeling can be in a curated ecosystem that is routinely conducting social experiments on its users.

“Usually, as we’ve seen, [... companies are] focused on making money. However, their profits are tightly linked to government policies. […] Now they’re gaining the wherewithal to fine-tune our political behavior—and with it the shape of American government—just by tweaking their algorithms.”

While O’Neil ultimately argues that there isn’t enough evidence that social media companies are intentionally and nefariously trying to shape government, the threat clearly lurks in this sentence. Worse yet, the clause starting with “just” reveals how easy it would be for social media companies to abuse the power they wield, if they had the motivation to do so.

“In a dream world, politicians would navigate countless such targeted safe zones so that they could tailor their pitch for every subgroup—without letting the others see it. One candidate could be many candidates, with each part of the electorate seeing only the parts they liked.”

This hypothetical may not be reality for politicians, but it can be the reality with technology. When everything is tailor-made to a particular audience, it’s easy for companies to only show the face they need the consumer to see. O’Neil forecasts a grim future if WMDs are allowed to persist without transparency and regulation.

“With political messaging, as with most WMDs, the heart of the problem is almost always the objective. Change that objective from leeching off people to helping them, and a WMD is disarmed– and can even become a force for good.”

Pinpointing the exact issue in the midst of a complex ecosystem of issues makes it seem solvable, and that’s exactly what O’Neil seems to be trying to leave readers with here: a sense that, while the situation is unfair and even in some cases dire, it can still be fixed with the right resolve. Intentions matter to a point, but the objective of the program is what leaves a lasting impact.

“Dismantling a WMD doesn’t always offer such obvious payoff. While more fairness and justice would of course benefit society as a whole, individual companies are not positioned to reap the rewards. For most of them, in fact, WMDs appear to be highly effective. Entire business models, such as for-profit universities and payday loans, are built upon them. And when a software program successfully targets people desperate enough to pay 18 percent a month, those raking in the profits think it’s working just fine.”

O’Neil positions an ideal society of justness and equality alongside the company’s goals, and since they are benefiting from the system as it is, they see no reason for change. Profit makes many large companies willfully blind.

“But human decision making, while often flawed, has one chief virtue. It can evolve. As human beings learn and adapt, we change, and so do our processes. Automated systems, by contrast, stay stuck in time until engineers dive into change them.”

By comparing humans to machines directly, O’Neil exposes one of the key flaws in WMDs: the inability to learn and incorporate feedback in meaningful ways to work towards improvement. Technology can only evolve if humans help it to change by inputting feedback and updating the automated systems.

“Sometimes, it is all too clear from the get-go that certain WMDs are only primitive tools, which hammer complexity into simplicity, making it easier for managers to fire groups of people or to offer discounts to others.”

This image of “hammering complexity into simplicity” demonstrates the powerful and destructive nature of WMDs. By forcing a layer of complexity on these decisions, WMDs make the decisions seem beyond understanding and change.

“After all, to police the WMDs, we need people with the skills to build them. Their research tools can replicate the immense scale of the WMDs and retrieve data sets large enough to reveal the imbalances and injustice embedded in the models.”

Throughout the book, O’Neil has demonstrated the ways WMDs police the average individual’s life, but here, she flips the script and suggests that society needs to be doing the same thing to the technology that governs it. Passivity is not an option, especially when a more equitable society is attainable if the minds creating the technology have the desire to create positive change.

“Finally, models that have a significant impact on our lives, including credit scores and e-scores, should be open and available to the public. Ideally, we could navigate them at the level of an app on our phones. In a tight month, for example, a consumer could use such an app to compare the impact of unpaid phone and electricity bills on her credit score and see how much a lower score would affect her plans to buy a car. The technology already exists. It’s only the will we’re lacking.”

By using a clear example to illustrate her idea, O’Neil shows how access to information can allow users to make more informed decisions that will ultimately lead to better outcomes. Without access to the exact workings of these models, a user cannot evaluate their options, and, therefore, is forced to guess what may be better for them in the long run.