Misinformation and What Platform Design Choices Actually Do

In 2021, Frances Haugen left Facebook and brought with her a large collection of internal documents. Those documents were reported on extensively by the Wall Street Journal and other outlets. One of the things those reports described was research that Facebook itself had apparently conducted, suggesting that its recommendation systems were amplifying divisive and emotionally provocative content. According to documents reported by the Wall Street Journal and others in 2021, the company was aware of this dynamic. I want to be careful here: I am describing what was reported, not what was proven in any independent audit. But the mechanism being described, that engagement-optimizing algorithms tend to amplify emotionally provocative content, is not a contested empirical claim. It follows directly from how the systems are built.

Most social media platform ranking algorithms optimize for engagement. Engagement is usually operationalized as some combination of clicks, shares, watch time, comments, and reactions. Content that generates strong emotional reactions gets more engagement. That content moves higher in feeds. It gets recommended more. It reaches more people who then engage with it, pushing it further. This is not a conspiracy. It is the predictable output of optimizing for a metric that correlates with, but is not identical to, what users actually value or what is true.

The distinction between correlation and identity is the key one. Engagement and value are correlated in the sense that people do engage more with things they find interesting or useful. But they are not the same thing. Outrage is engaging. False information that confirms existing beliefs is engaging. Conflict is engaging. None of those properties have any necessary relationship to accuracy or to the kind of experience users would endorse if you asked them at the end of the day whether that content improved their lives. The algorithm, optimizing for the proxy metric, does not make this distinction. It finds the content that drives clicks and surfaces it.

The IS argument I want to make is that this is a design outcome. Platform companies made choices about what to optimize for, how to measure it, and what trade-offs were acceptable. They made those choices in specific organizational contexts, under specific competitive pressures, with specific ownership structures and incentive systems. Some of those choices were deliberate. Others were inherited from earlier decisions about what engagement means and how to maximize it. But none of them were forced by physics or mathematics. They were decisions, and decisions have authors.

"Algorithmic governance" is a term that has appeared in IS and platform research to describe what happens when algorithms, not humans, mediate the rules under which content is seen, amplified, or suppressed. Möhlmann and colleagues have argued that on content platforms, algorithms do not just automate neutral processes. They govern how stakeholders interact and how conflicts among them are resolved. The decision about whose content is visible, whose is downranked, and what counts as a rule violation is not a neutral technical output. It is a governance decision that has been encoded into an algorithm and run at scale without a human making a judgment call on each individual case.

I wrote about this more directly in the context of platform governance and multi-sided markets, where the basic point is that platforms are not neutral intermediaries. They are governance structures with rules, and the rules are written to serve the platform's interests first. The misinformation problem is a specific instance of that more general point. When the rule is "maximize engagement," and when false or emotionally provocative content maximizes engagement more reliably than accurate content, the platform's rules are creating a structural advantage for that kind of content. The platform did not intend to create a misinformation machine. It intended to maximize engagement. The misinformation was an output of the rules it designed.

The response from platform companies to this framing has generally been some combination of: we label false information, we reduce its distribution, we remove the worst of it, and we cannot be responsible for all user-generated content. These responses are not dishonest. Labeling does reduce sharing of some false content, at least at the margins. But these interventions happen downstream of the core design choice, which is the engagement metric. You can label a false story and still surface it to millions of people before the label appears, because it generated early engagement before fact-checkers caught it. The downstream intervention treats a symptom. The upstream choice produces the condition.

The framing I find most useful is that platform design is social policy. The decision about what content to show people, in what order, and how to weight novelty against accuracy against social proof, is a decision with social consequences at enormous scale. Framing those decisions as purely technical, as choices about optimization targets and ranking parameters, misrepresents where the real policy choices lie. The algorithm does not make the policy. The people who chose what to optimize for, and the organizations that deployed the system without auditing its consequences, are making the policy. The algorithm just runs it faster than any human process could.

This matters for how we think about platform regulation. A lot of regulatory attention has focused on content moderation: what categories of content should be removed, how fast, with what appeal processes. That is a reasonable thing to regulate, but it is downstream of the structural problem. If the ranking algorithm continues to reward engagement over accuracy, then content moderation is a filter at the end of a pipe that is still pumping the wrong stuff. The more structurally significant regulatory question is what objectives platforms are allowed to optimize for, and whether engagement-maximizing systems need to be disclosed, audited, or modified in ways that account for their social effects.

I do not have a clean answer to what the right regulatory design looks like. That is genuinely hard, partly because engagement and value are correlated (not every engaging piece of content is harmful), and partly because platform companies have legitimate interests in keeping users on their apps. But the question only becomes answerable if you start from the right premise: that these are design choices made by people, with social consequences, and that "the algorithm decided" is not an explanation, any more than it was for the algorithmic bias cases I wrote about in the other post. The algorithm ran. People designed it. The consequences are on the design.