The Alignment Problem: Machine Learning and Human Values

Chapter 8 starts with a 2006 study undertaken by University of Michigan psychologist Felix Warneken, which demonstrated that toddlers can intuitively recognize and solve problems without direct encouragement or reward—a behavior that suggests a deep-seated propensity for cooperation and altruism. This insight, explored alongside Michael Tomasello, contrasts with other primates, which exhibit conditional and limited helping behaviors. Their findings underscore human uniqueness in social cognition, emphasizing humans’ natural inclination to collaborate and assist, an aspect that AI development might emulate through observing and learning from human actions.

In the 1990s, Stuart Russell, a UC Berkeley researcher, reflected on the uniformity of human walking patterns across different cultures and times. He suggested that the reason for this uniformity might not be simply imitation but a physical optimization of some sort, although the exact reason remained elusive. His inquiry led to the concept of inverse reinforcement learning, proposing a method to deduce the motivations behind observed behaviors, an approach that could advance AI development significantly by understanding and aligning machine actions with human intentions.

Inverse Reinforcement Learning (IRL) explores how actions can imply optimal behaviors under unknown reward conditions. Stuart Russell and Andrew Ng demonstrated IRL’s potential by modeling simple tasks and assuming perfect, non-random actions to infer straightforward goals. This concept evolved to model complex behaviors, like driving, where AI inferred human-like goals from observed actions without needing explicit instructions. Later, advancements enabled robots to perform intricate tasks by inferring intentions from demonstrations, suggesting that IRL can potentially teach machines to understand and replicate human values and complex behaviors beyond basic programmed instructions.

IRL infers complex goals within systems by observing expert demonstrations, although it depends on experts being available. The advantages of this technique are most significant in areas that are difficult to explicitly program. Examples include piloting helicopters for complex stunts and driving taxis where a human expert’s behavior provides the necessary data for learning. The challenge lies in whether systems can deduce reward functions solely from feedback without explicit demonstrations, which could revolutionize how machines align with human intentions, especially when direct programming of complex tasks is unfeasible. Researchers like Jan Leike, Paul Christiano, and Dario Amodei tested inverse reinforcement learning without direct demonstrations, using a virtual environment and feedback obtained through human-selected video clips. This method used the same reward system as in human behavior, aligning machine behavior with human judgments. The experiment extended to complex scenarios like the video game Enduro, where the system outperformed traditional methods.

In 2013, Dylan Hadfield-Menell joined Stuart Russell’s research lab at UC Berkeley to explore the ethical alignment of AI. They worked on developing cooperative inverse reinforcement learning (CIRL) to ensure AI systems align with human values, a shift toward collaborative human-AI interaction that built on Russell’s earlier work in inverse reinforcement learning. This approach sought to redefine AI development to prioritize human objectives.

Russel’s research lab, together with others working in the field, such as roboticist Anca Drăgan, introduced advanced cooperative frameworks in machine learning by borrowing insights from developmental psychology, education, and human-computer interaction. This approach recognizes that both humans and machines can benefit from understanding each other’s intentions, improving interaction. The methodology emphasizes mutual learning and adjustment, similar to pedagogical techniques used in human parenting and teaching.

To close Chapter 8, Christian discusses the integration of machine learning with human behavioral insights from fields like developmental psychology and education to enhance cooperation between humans and machines. This cooperative framework allows machines to interpret and learn from human actions and feedback more effectively. As machines become more embedded in daily lives, understanding and shaping their behavior becomes crucial. Christian advises readers to be more mindful of their actions online, as they should keep in mind that AI is constantly learning from their actions (such as browsing), therefore one should be aware of which behaviors should be reinforced.

Chapter 9 starts with the account of Stanislav Petrov, a Soviet officer supervising Oko (a satellite system used for warning), and the crisis he faced in 1983 when satellite warnings falsely indicated a US missile attack. Despite protocol urging immediate action, Petrov—who suspected a system error due to the unlikely scale of the attack—chose to report it as a false alarm. His intuition was correct; it was merely a malfunction caused by sunlight reflections, avoiding a potential retaliatory nuclear strike.

Despite the perceived reliability of warning systems like Oko, inconsistencies such as identifying random static as objects with high confidence have revealed flaws, specifically in deep-learning systems. These systems, trained only on defined categories, fail to recognize images that do not fall in any category (pixels without any meaning) effectively, pointing to a significant challenge in AI known as the “open category problem” (280). This issue was exemplified in a project led by Thomas Dietterich, where a system trained to identify specific insect species inaccurately classified non-insect objects in black and white, as the lack of color stalled the system’s identification process.

Christian discusses the fact that contemporary computer vision systems are highly specialized but suffer from a major limitation: They are trained to recognize only predefined categories, leading them to misclassify or overconfidently identify unfamiliar images. These systems lack a “none of the above” option and often make incorrect classifications with high confidence (281). Yarin Gal, a researcher of Oxford Applied and Theoretical Machine Learning Group, Oxford professor, and NASA scholar, emphasizes the importance of incorporating uncertainty into machine learning models. Researchers argue for the use of Bayesian principles to better manage and represent uncertainty. This approach could transform systems by allowing them to acknowledge when they do not recognize an input, enhancing reliability in practical applications like medicine and autonomous driving.

In 2017, an unidentified, unconscious man was brought into Jackson Memorial Hospital in Miami with a “Do Not Resuscitate” tattoo on his chest. Initially, Dr. Gregory Holt considered ignoring the tattoo due to the uncertainties it presented. As the man’s condition declined, ethical dilemmas intensified, leading to consultation with an ethics committee. They decided to honor the tattoo’s directive after verifying it mirrored an official request on file, and ultimately chose not to intervene further. The patient died, raising complex questions about courses of action that could be irreversible. AI safety research faces similar challenges in defining and managing the consequences of high-impact actions. Researchers like Stuart Armstrong and Victoria Krakovna are exploring how AI systems can avoid irreversible outcomes, even in trivial actions, by developing frameworks that prioritize minimizing potential negative impacts in uncertain situations.

In a 1960 article, MIT researcher Norbert Wiener articulated a foundational concept in AI safety, emphasizing the necessity of precise intent in machine programming. Wiener warned that once machines are activated, their operations become difficult to alter, making it crucial that their programmed purposes truly align with our intentions, not merely approximate them. This early insight into the alignment problem expresses a complex challenge: ensuring that AI’s actions strictly adhere to its human-defined goals. Additionally, this introduces the concept of “corrigibility”—the ability to correct or modify an AI’s course of action when necessary. Other researchers, such as those working at the Machine Intelligence Research Institute and the Future of Humanity Institute studied AI’s corrigibility by focusing on incentives. They experimented with balancing incentives for AI to allow goal modification, finding early strategies unsatisfactory but enlightening for future studies. They suggested that embracing uncertainty might be more effective than manipulating incentives, proposing a system that remains aware of its potential flaws and incompleteness. This idea aligns with concurrent findings from Berkeley, advocating for AI systems that prioritize human input.

Christian ends Chapter 9 with a comparison between Catholic theological debates and modern AI ethical considerations, pointing to an overall uncertainty in defining moral actions in both the Catholic and the machine contexts. The author draws an analogy between Catholic interpretations of sin, which struggle to balance strict adherence with more permissive attitudes, and modern dilemmas in AI on handling conflicting ethical theories. Just as theologians debated the sinfulness of actions like eating fish on Fridays, AI ethicists debate the parameters for machine actions under moral uncertainty. This historical context emphasizes the ongoing challenge of achieving ethical consensus, whether in religious doctrine or AI programming, suggesting that the complexities of guiding actions are rooted in current uncertain moral standards.