Chapter 5: Why Good Intentions Are Not Enough

The most common response to the concerns raised in the preceding chapters is some version of: the companies behind these systems are not trying to harm anyone. That observation is largely accurate, and also largely beside the point. Most of the harm that personalization infrastructure produces is not the product of malicious intent. It is the product of incentive structures that reliably reward behavior that produces harm as a side effect of pursuing metrics that look, in isolation, like reasonable measures of success. Understanding why good intentions fail systematically is more useful than debating whether the people involved are good or bad actors.

This chapter is addressed as much to people who build these systems as to people who use them, because the organizational dynamics that produce weaponized personalization are not unique to technology companies. They are recognizable patterns that appear whenever large organizations measure the wrong things, optimize aggressively for those measures, and then discover that the emergent behavior of their systems does not match the intentions of the people who designed them.

The Metric Problem

Organizations measure what they can, and optimize for what they measure. In the context of personalization, the metrics that became standard across the industry, time on platform, click-through rate, conversion rate, daily active users, average revenue per user, share the property of being both easily quantifiable and systematically misleading as proxies for user value. They measure activity rather than satisfaction, engagement rather than wellbeing, and transactions without any assessment of whether the person making them is better off for having done so.

When a metric becomes the primary target of optimization, systems naturally evolve toward whatever produces more of it, regardless of whether that thing is actually desirable. A feed optimization system targeting time on platform will discover, through ordinary experimentation, that emotionally activating content keeps people engaged longer than informative content. It will not discover this through anyone deciding to make the feed emotionally manipulative. It will discover it because that is what the data shows, and the system is designed to act on what the data shows. The harm is a predictable consequence of optimizing the wrong thing, not a product of anyone intending harm.

This is why appeals to intention miss the point. The teams running these experiments are generally trying to build better products. The engineers implementing the algorithms are generally trying to solve interesting technical problems. The executives approving the roadmaps are generally trying to grow businesses they believe in. None of that prevents the aggregate system from producing outcomes that a reasonable person, looking at them clearly, would find difficult to defend.

How Optimization Finds the Edges

There is a dynamic in machine learning systems that is worth understanding in some detail, because it is central to how personalization systems produce harmful outcomes without anyone explicitly designing them to do so. When you train a model to optimize a metric, the model will find the most efficient path to improving that metric given the data available to it. That path is not always the one the designers intended. Often it involves discovering patterns in human behavior that are real and exploitable but that no human analyst would have identified or chosen to exploit deliberately.

A recommendation system optimizing for watch time might discover that a certain sequence of content types produces a state of passive continued viewing in many users, not active enjoyment, but a kind of low-engagement continuation that looks the same in the data. A pricing model optimizing for conversion might discover that certain combinations of urgency signals and price presentation reliably increase purchase rates among people who later report regretting the purchase. A content ranking system optimizing for shares might discover that a particular kind of outrage-adjacent framing reliably outperforms informative framing on that metric, even when the informative content is more accurate and more useful.

In each case, the system has learned something true about human psychology and is exploiting it in the service of a metric. Nobody wrote code to manipulate people. The manipulation emerged from the optimization process finding the most efficient path to the number it was asked to improve. This is one reason why the standard industry response of "we didn't intend this" is simultaneously honest and inadequate. Intent does not constrain what optimization discovers.

The Fragmentation of Responsibility

Inside large technology organizations, the decisions that collectively produce personalization harm are distributed across many teams, roles, and time horizons in ways that make it genuinely difficult to locate where responsibility resides. The data science team running an A/B test is measuring whether version A or version B produces more of a specified metric, with no mandate to assess whether either version is good for the people experiencing it. The product team reviewing the results is evaluating whether the winning variant should be launched at scale, not evaluating its downstream effects on the people who will live inside it. The engineering team implementing the launch is solving a technical problem. What the variant does to user behavior over time sits outside the scope of any of these roles as they are typically defined.

Each of these teams is doing its job reasonably well by the standards it is being measured against. The harm emerges in the aggregate, from the combination of many locally reasonable decisions that nobody has evaluated as a whole. This is what makes it so persistent. The harm has no single point of origin: no specific decision that could be reversed, no one team that could be held accountable, no identifiable moment at which someone chose to produce the outcome that resulted. The harm is a property of the system, not of any individual decision within it, and fixing it requires changing the system rather than changing any specific decision.

Why Self-Regulation Has Not Been Sufficient

The technology industry has invested genuinely in mechanisms intended to prevent personalization harm from inside. Ethics review boards, trust and safety teams, responsible AI programs, and internal audit processes are all real commitments that real people spend significant time on. The fact that these efforts have not prevented the patterns described in the preceding chapters is not evidence that they are insincere. It is evidence of how powerful the structural incentives are that work against them.

An ethics review process that operates after a product decision has been made is reviewing a fait accompli. The teams that developed the product have already invested in it. The roadmap already incorporates it. The business case for it has already been approved. Reversing a decision at that stage carries organizational costs that a finding of ethical concern rarely outweighs unless the concern is very serious and very clear, which diffuse systemic harms often are not. A trust and safety team that is measured on response to reported incidents is structurally positioned to address harms that are visible and attributed, not harms that are emergent, distributed, and below the threshold of any individual complaint.

The deeper issue is that the incentives that produce personalization harm are not incidental features of how these businesses are organized. They are central to the economic model. A platform whose revenue depends on advertising needs engagement. Engagement is most reliably produced by the same mechanisms that Chapter 3 identified as harmful. Asking that platform to voluntarily undermine its own engagement metrics in the interest of user wellbeing is asking it to act against the economic logic that funds every other thing it does, including the ethics programs. The structural conflict cannot be resolved through goodwill alone.

What Actually Changes Behavior

Given all of this, the question worth asking is not whether the people building these systems have good intentions, but what kinds of external pressure or internal restructuring have actually produced meaningful change in how personalization systems operate. The honest answer is that the track record of voluntary internal reform is limited, and the changes that have had the most impact have generally come from combinations of regulatory pressure, competitive dynamics, and reputational consequences significant enough to reach decision-makers who control the metrics rather than people who are evaluated by them.

This does not mean that individual choices are irrelevant or that there is nothing for builders to do. Teams that treat "who could this harm and how" as a first-class question during product development, rather than an afterthought after launch, tend to produce meaningfully different outcomes. Organizations that build user welfare metrics alongside engagement metrics, and give those welfare metrics real organizational weight, tend to build different systems. Neither of these approaches is obscure or untested. The difficulty is sustaining the commitment to them under competitive pressure, when the things they protect do not show up anywhere on the revenue line.

Chapter 6 looks at what the landscape ahead actually looks like, where personalization is going as the underlying technology continues to improve, and what a more grounded set of expectations looks like for individuals, builders, and the regulatory environment that will increasingly shape what is permissible.