Noise: A Flaw in Human Judgment

Part 1 opens with the premise that “wherever there is judgment, there is noise – and more of it than you think” (21). Noise is present in infamous case studies in which two people convicted of the same crime receive wildly different punishments. It also exists in judgments involving apparently unique situations, like how to deal with a global pandemic.

According to the authors, most people would agree that two defendants convicted of the same crime should receive similar sentences, regardless of seemingly arbitrary factors such as the weather on the day of the trial or a local sports team’s performance at a recent game. However, both historically and in the present day, judges have enjoyed ample discretion in determining sentences; they even celebrate the fact that they take external factors into consideration. However, in the 1970s, judge and human rights advocate Marvin Frankel demonstrated that variability in sentencing was rampant and a means of perpetrating “arbitrary cruelties” (23). In his 1974 study of 50 judges, Frankel found that a heroin dealer, for example, could receive a sentence as varied as one-to-ten years in prison, depending on the inclinations of the judge who tried them. Frankel proposed that judges consider a detailed checklist and a numerical grading system to diminish the number of arbitrary factors that contribute to their decision.

Subsequent studies in 1977 and 1981 also showed the impact of noise on judges’ sentences. The studies demonstrated how irrelevant factors influence trial outcomes; for example, hungry judges are statistically tougher on defendants than judges who have returned from a food break.

Following Edward M. Kennedy’s 1975 sentencing reform legislation campaign, the US Sentencing Commission was formed and tasked with developing mandatory sentencing guidelines that would determine restricted ranges for the sentencing of particular crimes. The guidelines state that judges must take defendants’ crimes and criminal histories into account, and once these factors are considered, this narrows down judges’ sentencing options and cuts down the noise. While the guidelines succeeded in eliminating sentencing disparity, they received opposition from individuals like Yale Law professor Katie Stith and federal judge José Cabranes, who argued that they obstructed judges’ abilities to take the intricate particularities of every case into account, thus interfering with justice.

In 2005, the Supreme Court made the guidelines advisory rather than mandatory. While this decision found favor with 75 percent of federal judges, sentencing disparities increased significantly from 2005, according to a study by Harvard Law professor Crystal Yang. Before 2005, when the guidelines were mandatory, defendants sentenced by a particularly harsh judge were sentenced to an average of 2.8 months longer than average; however, once the guidelines were made advisory, the disparity between an average and harsh judge almost doubled. Moreover, the rate of African Americans receiving harsher sentences than white people committing the same crime increased after 2005.

The authors conclude that judgment is difficult because the world is complex and uncertain. However, the extent of divergence in sentencing can create “rampant injustice, high economical costs, and errors of many kinds” (31). They believe that noise can be reduced, and that some noise-reducing methods can remedy ills such as racism in sentencing.

An insurance company wanted to reduce inconsistency amongst those who made important financial decisions for the firm and found that noise was a significant factor in determining the magnitude of disparity. They undertook a noise audit to measure and resolve this.

The insurance company employs numerous underwriters and entrusts them to prepare quotes for clients according to a “Goldilocks zone” threshold, where the level premium is high enough to be advantageous to the company but low enough to be accepted by the client. Premiums that are outside of the Goldilocks zone cost the company money, as does the variability of judgments about how to define the Goldilocks zone. An element of chance is at play in matching an underwriter to a particular case, and the assignment of a colleague may produce a radically different result.

In the noise audit, employees were tasked with evaluating the same cases as their colleagues. While the company executives guessed that their employees’ evaluations would vary by 10 percent, the authors’ noise audit found that the median difference in underwriting was 55 percent. This disparity equated to a company loss of one hundred million dollars. It also showed that the customers who received wildly different premiums depending on the underwriter who got their case were “signed up for a […] lottery without their consent” (37).

While the authors expected the presence of disruptive noise in the organizations they studied, they were surprised that company executives were under the illusion that such high levels of disparity of judgment were not present in their workforce. This is because the workforce maintains “the illusion of agreement while in fact disagreeing in their daily professional judgments” (41). The employees possess a mistaken bias that others view the world and judge it in the same manner as they do. Moreover, the widespread discomfort with disagreement causes companies to minimize the occasions where exposure to conflicts in judgment occurs. This false illusion of harmony masks the variability in the employees’ decision-making.

While noise and unwanted variability are easy to define and measure in recurrent decisions such as sentencing defendants or diagnosing patients, it is often more difficult to identify in “a category of judgments that we call singular decisions” (46).

Unlike recurrent decisions, singular decisions are only made once and often by a group of people who have never worked together. Therefore, there is no standard course of action in many important political decisions such as going to war or dealing with a pandemic; the same is true of personal decisions, like buying a house or proposing marriage.

In the academic realm, approaches to singular and recurrent decisions are different. While analyses of recurrent decisions have a statistical bias and are the province of social scientists, it is historians who are appointed to investigate singular decisions in hindsight and focus on the causes behind what happened.

The definition of noise is different as it applies to singular decisions, and therefore it must be measured differently. Individuals still must account for the fact that two people in the same unique situation would behave differently, and even the decision maker should be aware that they may have reached different conclusions if some irrelevant variables in the situation had been otherwise. This becomes evident when considering how different countries handled the unique situation of the COVID-19 pandemic. According to the authors, the variation of responses provides evidence of noise in different countries’ strategies. The authors write that “from the perspective of noise reduction, a singular decision is a recurrent decision that happens only once. Whether you make a decision only once or a hundred times, your goal should be to make it in a way that reduces both bias and noise” (50). For that reason, the noise-reduction practices applied to recurrent decisions can be useful for singular decisions, too.