AI Alignment Has a Stakeholder Problem
Alignment is usually described as alignment with human values. The papers that built the method say it aligned models to a small group of people, and IS research on trust shows that creator, user and system pull in different directions.
The survey by Ji et al. (2023) opens with a definition that I see everywhere. "AI alignment aims to make AI systems behave in line with human intentions and values." Ask an IS researcher what an enterprise system is aligned with, and the answer is usually a list of stakeholders with different interests. I think the second answer is closer to the truth for AI as well, and the most careful alignment papers say so themselves.
Ouyang et al. (2022), the InstructGPT paper, is unusually direct about whose preferences went into the model. Their procedure "aligns the behavior of GPT-3 to the stated preferences of a specific group of people (mostly our labelers and researchers)". They "hired about 40 contractors, guided by their performance on a screening test", and those people "disagree with each other on many examples". The authors report "the inter-labeler agreement to be about 73%." They also write that "we are aligning to our preferences, as the researchers designing this study (and thus by proxy to our broader research organization, OpenAI)". I wrote about the labeler side of this in an earlier note on RLHF and structuration. Here I want to stay with the list of people they did not align to.
That list is the part an IS reader should notice. Ouyang and colleagues name "the organization training the model, the customers using the model to develop products, the end users of these products, and the broader population who may be directly or indirectly affected". They give an example of a conflict. For instance, "a customer may want a model that maximizes the amount of time a user spends on their platform, which is not necessarily what end-users want." And they draw the conclusion that "it is impossible that one can train a system that is aligned to everyone's preferences at once, or where everyone would endorse the tradeoffs."
Casper et al. (2023) state the technical side of the same point. "A single reward function cannot represent a diverse society of humans." And "Attempting to condense feedback from a variety of humans into a single reward model without taking these differences into account is thus a fundamentally misspecified problem." Bai et al. (2022) say from the design side that "when developing and deploying a general AI system, we cannot avoid choosing some set of principles to govern it, even if they remain hidden or implicit." They describe their own principles as "chosen in a fairly ad hoc and iterative way for research purposes", and they think that in the future such principles "should be redeveloped and refined by a larger set of stakeholders".
Gabriel (2020) frames the normative question, asking how we decide which principles to encode, and "who has the right to make these decisions", "given that we live in a pluralistic world that is full of competing conceptions of value". Hadfield-Menell and Hadfield (2018) name the people whose interests are at stake when they describe the gap between a reward function and "what relevant humans (the designer, the user, others affected by the agent's behavior) actually value".
Saffarizadeh, Keil and Maruping (2024) bring this into IS research. Their starting point is that earlier work "has not addressed the possibility that relevant stakeholders may hold different goals and values", and they describe their own contribution as "proposing a stakeholder-centric approach to AI alignment". Using the agentic IS framework, they identify "creator-AI alignment, user-AI alignment, and AI internal-alignment". They define creator-based steerability as "the perceived ability of an AI agent to be directed to behave in a manner that aligns with its creator's goals and values", and user-based steerability as "the perceived ability of an AI agent to be directed to behave in a manner desired by its user". Across "four experiments" with "a total of 1,140 participants", they find that "creator-based steerability amplifies trust transference from AI creator to AI system, while user-based steerability and autonomy diminish it." They also report that "people are more likely to trust an AI agent with high user-based steerability, low creator-based steerability, and low autonomy."
What I take from this is that alignment is not one relationship. It is at least three, and they can pull against each other. The authors say directly that "there may be important trade-offs associated with optimizing for creator-AI agent alignment and optimizing for user-AI agent alignment." Their call is to "theorize from a triadic view encompassing the user, the AI system, and its creator." I would go one step further, following the stakeholder list in Ouyang and colleagues. The triad leaves out the paying customer who builds a product on the model, and the people affected by the model who never touch it. Saffarizadeh and colleagues name this limit themselves, noting that "factors related to the AI agent's alignment with other stakeholders such as the government and political parties" may also matter.
For IS this is a familiar move, since stakeholder analysis is old ground for us. What is new is where the conflict sits. It is built into the model during training, before any organization deploys it. So the question of who the model is aligned to has to be asked twice, once of the lab that trains the model and once of the organization that deploys it. The second question is the subject of the note on alignment as an organizational promise.