When the Algorithm Fails, Nobody Takes the Call

Boeing's MCAS system took two aircraft down in 2018 and 2019, killing 346 people. The design logic was that the flight control software would prevent aerodynamic stall without requiring pilots to know it was running. The pilots, in other words, were principals who had delegated control to an algorithmic agent they did not know existed, let alone know how to override. When MCAS activated incorrectly on Lion Air Flight 610, the pilots pushed the nose back up. MCAS pushed it back down. They pushed back up again. MCAS responded again. The system was acting on faulty sensor data and the humans trying to override it did not understand what they were fighting. The information asymmetry was catastrophic. The agent had information the principal did not.

Jensen and Meckling (1976) laid out the structure of this problem before computers were doing anything close to what MCAS was doing. Their core argument is that when a principal delegates work to an agent, three problems arise almost automatically. First, goal conflict: the agent's interests may not align with the principal's. Second, information asymmetry: the agent will usually know more about what they are doing and why than the principal can observe. Third, agency costs: the principal must spend resources monitoring the agent's behavior, the agent may spend resources signaling their trustworthiness, and there is always a residual loss from the imperfect alignment that remains after both of those efforts.

Eisenhardt (1989) extended this to organizational contexts and added a distinction that became foundational for IS governance research. When the principal can observe the agent's behavior, behavior-based contracts make sense: pay the agent for doing the right things. When behavior is hard to observe but outcomes are measurable, outcome-based contracts make more sense: pay the agent for producing the right results. IT outsourcing is the classic IS application of this framework. The vendor (agent) has deep technical knowledge the client (principal) does not have. The principal cannot directly observe whether the vendor's engineers are doing good work. The principal can often observe outcomes: system uptime, response time, deliverable quality. So IT contracts tend to be outcome-based, and the monitoring architecture is built around measurable outputs rather than behavioral surveillance.

The theories.html from my study hub is clear about what agency theory is for: "Use it for outsourcing, platform governance, monitoring, incentives, and AI delegation when control and responsibility are unclear." The key mechanism is: "Monitoring, incentives, rules, and accountability reduce misalignment and risk." This is clean and tractable when the agent is a human organization. When the agent is an algorithm, something fundamental breaks.

Consider the four pieces of the standard agency response to misalignment: monitoring, incentives, rules, and accountability. Monitoring: you can log every output an algorithm produces, but you usually cannot observe the process that generated it. The MCAS software was doing something, but the logs were not surfaced to pilots in real time, and the pilots were not trained to interpret them. The algorithm was doing something, but the "monitoring" available to the principals was a display that said "nose angle" without explaining why the nose was going down. Incentives: you cannot give an algorithm a bonus for good behavior or a fine for bad behavior. Incentives are a mechanism for aligning interests, and an algorithm has no interests to align. Rules: you can constrain an algorithm with rules, but rules embedded in software are opaque to the principals who are supposed to be governed by them, and rules that interact with real-world complexity produce emergent behaviors the rule-writers did not anticipate. Accountability: who is accountable when MCAS kills 346 people? Boeing, certainly. The FAA regulators who certified the system. The engineers who designed it. But not the algorithm itself, because it cannot be held accountable in any meaningful sense.

This is what I find most interesting about the current AI governance moment. The standard toolkit of agency theory, the one Eisenhardt (1989) applied to IT outsourcing and that Jensen and Meckling (1976) developed for organizational control, depends on the agent being an entity with interests, observable behavior, and capacity for accountability. An algorithm in a production system has none of these properties cleanly. The agency problem with AI is not that the algorithm's interests diverge from the principal's. It is that the algorithm has no interests, generates information asymmetry by design (the black box), and makes the accountability question unanswerable with traditional tools.

Amazon discovered this with their recruiting algorithm, which became widely reported in 2018. The system was trained on ten years of historical resumes. The principal (Amazon's recruiting team) wanted a fair, efficient screening process. The agent (the algorithm) optimized for historical patterns, which reflected the industry's past gender imbalances. The agent did exactly what it was designed to do: learn from the data it was given. The principal's stated goal, fair hiring, was not in the training objective. The information asymmetry was built in from the start, because nobody could easily see what features the algorithm was using to rank candidates until the outputs were compared across categories. By then, the bias was already embedded in months of screening decisions.

The governance frameworks my study notes describe for IT systems are all calibrated for human agents. Monitoring assumes you can observe or audit behavior. Outcome-based contracts assume you can measure the outcomes accurately and attribute them to the agent's behavior. Residual loss assumes there is a bounded gap between what the principal wants and what the agent delivers. With AI systems, the audit is incomplete because interpretability is limited, the outcome attribution is complicated because the algorithm operates in interaction with human action, and the residual loss may not be bounded at all.

I think this is why most AI governance frameworks feel like wish lists. They name principles, transparency, fairness, accountability, explainability, without specifying who the principal is, what monitoring mechanism is available, and what happens when the algorithm fails in ways that produce diffuse harm rather than a single attributed event. A governance framework that does not name the principal-agent relationship is not really a framework. It is a list of values.

The connection to what I wrote about delegation and the failure of use-based measurement is direct. When an AI system acts on behalf of an organization, the relationship is a principal-agent relationship, but the tools for managing that relationship were not built for algorithmic agents. The delegation is real. The management infrastructure is not. And as I wrote about platform governance and layered agency problems, the situation gets more complicated when multiple principals are involved. Platform owners have principals in developers, users, regulators, and advertisers. AI systems deployed on platforms add another agent with no interests and an opaque process into a principal stack that was already hard to manage.

My read is that the most important unanswered question in AI governance is not "what values should the algorithm have?" It is: what does the monitoring mechanism look like when the agent is a black box, the outcomes are diffuse, and the accountability chain runs from algorithm to engineer to product team to organization to society without a clear stopping point? Jensen and Meckling gave us the cost accounting of agency. The field has not yet given us the equivalent framework for a world where the most consequential agents cannot be held responsible for anything.