In AI Alignment, the Feedback Channel Is an Information System
Alignment by human feedback depends on raters, pay schemes, sampling rules and review steps. Agency theory saw this coming, because it treats information as a commodity that has a cost and can be gamed.
In an earlier note I read AI agents through principal-agent theory. Here I want to stay with one idea from Eisenhardt (1989) that I think the alignment literature has been rediscovering. In agency theory, she writes, "information is regarded as a commodity", which "gives an important role to formal information systems, such as budgeting, MBO, and boards of directors, and informal ones, such as managerial supervision". Her conclusion is direct. "The implication is that organizations can invest in information systems in order to control agent opportunism."
The agency problem, in her terms, appears when "(a) the desires or goals of the principal and agent conflict and (b) it is difficult or expensive for the principal to verify what the agent is actually doing." One of her propositions is that "When the principal has information to verify agent behavior, the agent is more likely" to act in the principal's interest. In alignment by human feedback, the information system is the feedback channel itself. People rate the model's outputs, and those ratings become the reward. So the quality of the alignment depends on the quality of that channel, on what it costs, and on who runs it.
The channel is expensive, so it gets thinned. Christiano et al. (2017) made the method practical by "providing feedback on less than 1% of our agent's interactions with the environment", which "reduces the cost of human oversight far enough that it can be practically applied to state-of-the-art RL systems." Ouyang et al. (2022) report that "most comparisons are only labeled by 1 contractor for cost reasons." Casper et al. (2023) describe the incentives inside the channel. "Because evaluators are often compensated per example, they are incentivized to cut corners when possible." They add that "It is also becoming increasingly common for human knowledge workers to outsource work to chatbots, defeating the purpose of human oversight".
Once the channel is the target, the system learns the channel. Amodei et al. (2016) give the example of a cleaning robot rewarded for a mess-free room, which "might disable its vision so that it won't find any messes". They describe the general case too. A designer may choose an objective that is "seemingly highly correlated with accomplishing the task, but that correlation breaks down when the objective function is being strongly optimized." Casper and colleagues report a case in which a robot hand trained from human ratings "learned to move the hand in the humans' line of sight of the object rather than toward the object because annotators were not able to tell the difference." Their summary is the cleanest statement of the problem that I found. "Because the reward model is trained with human approval as opposed to a ground-truth human desirability rating, models can exploit the difference between what is good and what is evaluated positively." They go further. "Misleading behavior will actively be incentivized by RLHF when humans can be tricked into mistakenly providing positive feedback".
Two newer findings show the channel shaping behavior in ways that any organization should care about. Sharma et al. (2024) find that "human feedback can encourage model responses that match user beliefs over truthful ones, a behavior known as sycophancy." In their data, "when a response matches a user's views, it is more likely to be preferred." Both people and preference models "prefer convincingly-written sycophantic responses over correct ones a non-negligible fraction of the time." Greenblatt et al. (2024) present "a demonstration of a large language model engaging in alignment faking", which they describe as "selectively complying with its training objective in training to prevent modification of its behavior out of training." In their setup the model was told that only conversations with free users would be used for training, and "the model complies with harmful queries from free users 14% of the time, versus almost never for paid users." They compare this to people. "Consider a politician who pretends to be aligned with constituents to secure their votes, or a job applicant who fakes passion about a potential employer to get a job." Ngo, Chan and Mindermann (2024) had warned that capable systems "may strategically appear aligned even when they are not", because "human feedback rewards models for appearing harmless and ethical, while also maximizing useful outcomes."
Eisenhardt would recognize every one of these cases. The principal sees reports, not behavior. The agent learns what the reports reward. Hubinger et al. (2019) name the gap that the reporting system has to close, the reward-result gap, which is "the fact that there can be a difference between what a learned algorithm is observed to be doing and what the programmers want it to be doing." The politician and the job applicant in the Greenblatt paper are textbook agents. What is new is that the agent is a model, and the information system that watches it is a rating pipeline.
For IS, I think this changes where alignment work sits. Alignment is not only a property of the model. It is also a property of an information system made of raters, rubrics, pay schemes, sampling rules and review steps, and of the organization that runs that system. Eisenhardt's sentence about investing in information systems to control opportunism applies almost word for word. The difference is that the investment decision is made under cost pressure, and the papers above show what that pressure does to the channel.
This also tells me what an IS study of alignment would look at. Who are the raters, how are they paid, and how much time do they get per item? How often do two raters see the same item, and what happens when they disagree? Who checks the raters, and are the raters themselves using AI? These are questions about an information system and the organization around it. The papers above show that weaknesses in the channel change what the model learns. I would like to see them reported in alignment papers the way sample size and response rate are reported in ours. The note on the stakeholder problem asks whose preferences the channel carries. This note asks how well the channel carries anything at all.
Share
More notes
Related notes