When AI Gets It Wrong, Who Is Responsible?

A credit applicant is denied a loan. The bank's AI model scored them as high-risk based on patterns in their financial history, their zip code, and variables the applicant cannot see and was never told about. The applicant asks why they were denied. The bank's customer service representative cannot explain the model's decision. The model's vendor says the model performed as designed. The manager who approved the deployment of the model moved to a different role six months ago. The data science team that built it works for a third party.

Who is responsible?

This is not a hypothetical edge case. It is the standard operating condition for algorithmic decision-making in lending, hiring, insurance pricing, healthcare triage, and benefits eligibility. And it represents an accountability gap that is genuinely new in organizational life. Traditional organizational accountability was designed for human decision-making chains. When a loan officer denied your application, there was a person who made that decision, who worked for an organization with policies governing how decisions should be made, who operated within a regulatory framework requiring disclosed criteria. You could dispute the decision. There was a chain of accountability from the decision back to responsible parties.

AI systems break that chain in several ways simultaneously. The decision logic is often opaque, embedded in model weights that cannot be summarized as human-readable rules. The training data that shaped the model's behavior may have been collected years earlier, by a different team, from sources with their own biases and gaps. The model may have been built by one organization and deployed by another. The deployment context may differ significantly from the training context in ways that were not anticipated. And the model makes decisions continuously, at scale, without case-by-case human review. No single human made the decision that affected any particular applicant. The decision was made by a system that was designed, trained, validated, deployed, and monitored by a collection of humans operating at different points in time.

The legal frameworks are catching up, but slowly. The EU AI Act, which entered into force in August 2024, is the most comprehensive regulatory response so far. It creates a mandatory conformity assessment regime for high-risk AI systems, which explicitly includes AI used in employment, credit scoring, access to essential services, and several other categories. High-risk AI systems under the Act require technical documentation showing how the system works, evidence that bias and accuracy have been evaluated, human oversight mechanisms that allow humans to override or correct AI decisions, and registration in an EU database. The extraterritorial reach of the Act means that organizations outside the EU that deploy AI affecting EU persons in high-risk categories are subject to these requirements. I wrote about this in more detail in my post on the EU AI Act's IS implications, but the accountability dimension deserves its own treatment.

For IS researchers, the interesting observation is that the EU AI Act creates conformity assessment requirements that are fundamentally organizational governance requirements dressed in regulatory language. Saying a high-risk AI system requires "human oversight mechanisms" is saying that the organization must design processes and assign roles that allow humans to review, override, and correct AI decisions in practice, not just on paper. That is an organizational design problem. Saying the system requires technical documentation of its logic and training data is saying someone in the organization must maintain that documentation, keep it current as models are retrained and updated, and make it available when required. That is an information governance problem. Saying accuracy and bias must be evaluated is saying someone is responsible for ongoing monitoring, that there are metrics and thresholds and escalation paths when things drift out of range. That is an operational governance problem.

The accountability gap shows up clearly when you look at how organizations actually govern AI deployments through the lens of Feldman and Pentland's work on organizational routines. Their distinction between the ostensive routine (how a process is documented and supposed to work) and the performative routine (how it actually works in practice) applies directly to AI governance. Most organizations have ostensive AI governance: an ethics policy, a model risk management framework, a responsible AI statement on the website. The performative AI governance is often thinner. The team that deployed the model has moved on to the next project. The monitoring dashboard exists but nobody reviews it regularly. The human escalation path is documented but the humans on the receiving end of escalations have not been trained on what to do with them. The ostensive and performative routines diverge, and the divergence is invisible until something goes wrong. I should note that my study-hub files do not contain the Feldman and Pentland (2003) paper directly; I am drawing on the broader IS and organization theory literature where this framework is well-established.

Gartner has published commentary on AI governance and what they describe as the organizational readiness gap between AI deployment and AI governance capability. You can explore their publicly available research and press releases at https://www.gartner.com/en/newsroom. The general direction of that commentary, as I understand it from publicly available coverage, is that organizations are deploying AI faster than they are building the governance structures to manage it responsibly. I would not attribute specific numbers or percentages to Gartner without a current subscription-level source, but the directional claim is consistent with what IS researchers are observing in fieldwork and organizational case studies.

The vendor accountability question is particularly thorny. When an organization deploys a third-party AI model, they are making decisions based on a system they did not build and may not fully understand. Vendors typically disclaim liability for decisions made using their models. The organization deploying the model is the one making decisions. But the organization did not design the model, cannot inspect its weights, and may not have the technical capacity to fully evaluate the model's behavior on their specific population. The result is a split accountability structure where the vendor has the technical knowledge and the deploying organization has the legal and ethical accountability. That split creates incentives to avoid deep scrutiny: the vendor has no liability and the deploying organization may lack the capability to scrutinize even if it wanted to.

The principal-agent framing helps here. The organization deploying an AI model is in a principal role relative to the AI vendor as agent. But the information asymmetry runs strongly in the agent's direction. The vendor knows far more about how the model works, what its failure modes are, and what its performance limitations are than the deploying organization can reasonably evaluate from standard validation documentation. That asymmetry is not unique to AI, every enterprise software relationship has some version of it. But the AI case is more acute because the decisions being made by the system are consequential, often irreversible from the affected person's perspective, and difficult to explain even with full technical access.

The question I keep returning to is what accountability actually requires in this context. It is not enough to have a policy that says humans are responsible for AI decisions. Accountability requires that there be specific identifiable humans who can answer for specific decisions, who have the information needed to evaluate those decisions, and who have the authority and responsibility to correct them when they are wrong. Building that into an AI deployment is an organizational design challenge. It means defining roles, designing information flows, creating escalation paths, and training people. It means treating AI governance as an ongoing operational function, not a one-time compliance exercise.