Noise: A Flaw in Human Judgment

Inherent to the concept of judgment is the idea that a decision of agreed-upon ethicalness can be reached, rather than some whimsical response determined by opinion or taste. However, matters of judgment occupy an uneasy middle zone between fact and opinion, defined by “the expectation of bounded disagreement” (56).

Judges are prone to disagree on difficult problems, rather than those of obvious morality. (Here, the authors begin to use the term “judge” to describe any decision-maker, and not just judges in the court of law). To prove this point, the authors urge the reader to evaluate a fictional CEO candidate on a scale of zero-to-100, according to the likelihood of their ability to keep their job after two years. The fictional candidate, Michael Gambardi, has a history of astonishing accomplishments and failures alike, making it nearly impossible to judge his future performance.

The authors then define the steps in the reader’s process of reaching a number. They draw attention to the matter of “selective attention and selective recall” regarding the number of irrelevant personal details about Gambardi, which in themselves can create variance in different judges (58). The reader is likely to integrate these cues to reach an informal evaluation of Gambardi’s prospects and then convert the impression into a number on a scale between zero and 100. The authors speculate that some number likely came to the reader’s mind, and if that number did not feel entirely right, the reader would adjust it up or down. The reader would have settled on a final number according to their “internal signal of judgment completion”—a feeling that signaled that they had satisfactorily completed the task (61). In both predictive judgments, where the future outcome is unknown, and evaluative judgments, where a situation must be measured and assigned a consequence, decision-makers develop justifications for their hunches and strive for a sense of internal coherence.

The authors argue that the fact of noise, which produces disparity in different people’s predictive judgments, indicates error which can have harmful consequences for people’s lives. Moreover, people who are affected by these judgments expect that they are being evaluated by a system and not by the whims of individuals. Thus, the presence of unwanted variance damages the system’s credibility.

Noise can be measured using the concept of standard deviation. The more divergent the numbers, the noisier and less accurate the predictions. There are two error equations, the first of which decomposes the error into a single measurement: “error in a single measurement = Bias + Noisy Error” (78). The second is Mean Squared Error (MSE) which “can be shown to be equal to the sum of the squares of bias and noise” and is “Overall Error (MSE) = Bias² + Noise²” (78). While bias and noise act independently, they are equally weighted in determining overall error and are therefore interchangeable when calculating these equations. However, while bias gets more attention, there are many situations where noise is more responsible for the commission of large errors.

The authors posit that a maxim of good decision-making is to keep values separate from facts and preferences. Both predictive and evaluative judgments become important in decisions, as the likely consequences of actions are considered. The authors argue that “predictive judgments will be improved by procedures that reduce noise, as long as they do not increase bias to a larger extent” (84).

Noise audits, in which the same group of people judge several cases, allow for a more detailed analysis of system noise. This was the case in 1981, during a detailed noise audit of sentencing by federal judges to determine the factors behind disparity. 208 federal judges were presented with 16 hypothetical cases in which a defendant had been found guilty and was to be sentenced. The prison sentences dispensed by the judges for each case varied greatly, especially as “there is no objective way to determine what the ‘true value’ of a sentence is for a particular case” (88). The study showed that while the mean prison sentence was seven years, the standard deviation around the mean was 3.4 years. While such disparity is disturbing, the authors point out that in real life cases the standard deviation is likely to vary even more, as the hypothetical cases were easy to compare and presented successively.

Differences in the sentences passed by judges, who could be either harsh, lenient, or somewhere in between, was determined by criteria such as background, life experience, political opinion, and bias. Similarly, the judges’ propensity to think of sentences in terms of either rehabilitation or deterrence determined the length of their prison sentences. Importantly, none of these factors has anything to do with the crime or the defendant.

However, the data also shows pattern errors committed by pattern noise, whereby individual judges are shown to be uncharacteristically harsh or lenient in sentencing particular cases. For example, a judge could be harsh in general but lenient towards sentencing white-collar criminals. Or, they could be informed by unconscious biases, expressing tolerance towards a defendant who looks like their daughter, for example. The authors stipulate that “whatever their origin, these patterns are not mere chance: we would expect them to recur if the judge saw the same case again” (94). Because pattern noise is tricky to forecast, it adds a further level of uncertainty to the already unpredictable practice of sentencing. Still, it is inevitable wherever decisions are made. There is also the possibility of random error, whereby the judges’ sentences may have varied if they judged them on an alternative occasion. For example, judges tend to be more lenient when factors external to the case put them in a better mood. The authors call this “within-person variability” and state that it produces “occasion noise” (96).

Variability in sports performance, even amongst the same athletes, is expected, as is the variability of corporeal processes such as heartrate. However, the authors argue that observing the variability of a person’s mind is more difficult. People’s opinions change for no reason without the input of new information. This is even the case with professionals practiced at making similar decisions. Occasion noise is a key factor in this.

By using big data and econometric models, it is possible to check where “decisions were influenced by occasion-specific, irrelevant factors” such as temperature or time of day (101). Another way to find out occasion-specific decision-making would be to follow psychologists Edward Vul and Harold Pashler’s example and ask people the same question twice. Vul and Pashler found that the average of the two answers was normally more accurate than either single answer. This is derived from the “wisdom of crowds” effect. The variation in an individual’s answer to the same question was the effect of “the crowd within” (103). The benefits of asking a second time rose when the time between interrogations was greater. The benefit increases even further when the test subject seeks the opinion of outsiders who are not biased by the same occasion noise.

Of all the types of occasion noise, mood has the most important impact on decisions because it affects the information a person takes from their environment and which memories they recall. A good mood can be a mixed blessing: It makes negotiations with others go smoother, but it also makes people more prone to accepting first impressions and biases without challenging them. A bad mood, on the other hand, sharpens one’s judgment. A person’s mood in a particular moment will influence their approach to a complex judgment problem. However, variability in judgment is also determined by the “moment-to-moment variability” in the functioning of individual brains (113). This variable shows that occasion noise cannot be eliminated.

Although it is difficult to measure how large occasion noise is compared to system noise, the authors agree that its effects are generally smaller. Thus, while a person’s mind is not the same from day to day, or hour to hour, it is generally more similar to itself on a previous occasion than to another person’s mind.

Group decision-making can contribute to noise due to the consideration of irrelevant factors such as interpersonal dynamics. People may be influenced by which person speaks first and last, or by what their colleagues are wearing. As researcher Matthew Salganik discovered, “popularity is self-reinforcing” and can lead to copy-catting in the making of decisions (117).

This is the case with matters of taste such as music downloads, but also in important political phenomena such as referenda. Here, too, popularity and the enthusiasm generated on the first day of a campaign reinforces itself and influences outcomes. The authors emphasize that “independence is a prerequisite for the wisdom of crowds. If people are not making their own judgments and are relying instead on what other people think, crowds might not be so wise after all” (121). People’s propensity to listen to others and change their opinions to match theirs depends on how much they trust the person who speaks first and on the lack of reasons to suspect that they are wrong. This produces an “informational cascade,” which is a direct factor in the production of noise (123). While it is not always illogical to follow the opinions of others, especially if one lacks a strong conviction of one’s own, this neglects the fact that the others are also part of the informational cascade and therefore not following their own judgment. Thus, their judgment does not reflect “collective wisdom” so much as the “initial views of just a few people” (124). The social pressure to be seen as a team player also contributes to the informational cascade, as people want to maintain the group’s good opinion of them.

The informational cascade is amplified when people must make decisions by jury, as the effect of speaking with each other causes people to adopt a more extreme valuation than they might have originally adopted. The authors’ found that in the case of juries, deliberation produced increased noise and, with it, the increase of effects such as social influence. Thus, when the response of the median member of a six-group jury to a case was lenient or severe, the rest of the jury modifies its position toward one extreme.