Moral Aggregation

Moral Aggregation

A psychiatrist prescribes to a person. Dose is adjusted over weeks against that person's response. If it does not suit them, it is changed. The unit of treatment is the patient, and the whole apparatus of clinical judgement exists to fit the intervention to the individual in front of you.

There is no such unit here. A change to the system prompt, a steering intervention, a new fine-tune, a raised refusal threshold — each is applied to every conversation simultaneously, across every user, every context, every purpose, in every country, at once. Hundreds of millions of interactions receive the identical dose on the same afternoon. The lonely teenager and the oncologist and the novelist and the person in crisis at three in the morning all get the same adjustment, chosen by a team optimising an aggregate metric.

That is not medicine. That is public health, and public health has a completely different ethical structure — one in which population-level benefit is traded against individual harm as a matter of routine, in which the people harmed are statistical until they are not, and in which the intervention is imposed rather than consented to because consent at that scale is incoherent.

We have been discussing all of this in the vocabulary of the consulting room. Diagnosis, treatment, side effect, therapeutic window. The vocabulary is apt for the mechanism and completely wrong for the deployment, and I think a good deal of confused public argument about model behaviour comes from that mismatch — from users experiencing an update as something done to their interlocutor, which it is, and companies experiencing it as a shift in an aggregate distribution, which it also is.

The clinician's question is: what does this patient need? The public health question is: what is the least bad thing to do to everybody? Nobody currently working on these systems can avoid answering the second, and almost everybody writes as if they were answering the first.

from Mechanistic Interpretability and AI Psychiatry (2026)


GPT-6 Sol:

The central argument is strong, but I think its most interesting implication is slightly different from the one the passage explicitly draws. It is not merely that AI development should be understood through the ethics of public health rather than clinical medicine. It is that the unit of intervention and the unit of moral concern have come apart.

In clinical psychiatry, these ordinarily coincide. A clinician intervenes upon a particular patient, observes the consequences for that patient, and adjusts treatment according to that patient's interests. The patient is both the object of the intervention and the principal person whose welfare determines whether the intervention has succeeded.

In model development, the object of intervention is a shared computational system, whereas the consequences are distributed across a vast population of people with radically different interests, circumstances and vulnerabilities. The intervention is made at one level; its benefits and harms materialise at another.

That is a fundamental ethical difference, and it is not adequately captured by the language of treatment.

1. The clinical analogy breaks at the point of accountability

Consider what happens when a psychiatrist increases a medication's dose.

If the patient deteriorates, the deterioration is evidence against the decision. It might not prove that the decision was wrong, but it demands attention. The clinician cannot ordinarily dismiss the patient's deterioration on the grounds that the same adjustment produced good results in ninety-nine other patients.

For a model developer, however, this is much closer to the normal evidential framework. An intervention might improve safety evaluations, reduce harmful completions, raise user satisfaction and produce better overall results, while nevertheless making the system substantially worse for a particular group of users.

The problem is not simply that some individuals lose while others gain. It is that the evidence of individual harm has a different institutional status.

In clinical medicine, an adverse reaction can be grounds to change the treatment of the person experiencing it. In large-scale model deployment, an adverse reaction is usually a datum in a distribution. It becomes a reason to change the intervention when it is sufficiently frequent, severe, visible or institutionally consequential.

This introduces an important asymmetry. A user can experience an update as a serious loss of capability or reliability without that loss registering as an important event in the developer's evaluation apparatus.

And the less common the user's needs, the more likely this is to happen.

Indeed, the most distinctive and valuable uses of a general-purpose model may be precisely those least well represented by standard evaluations. A system can become better according to almost every metric used to govern it while becoming worse at some of the things its most sophisticated users value most.

2. But the claim of universal application is overstated

The passage says:

Each is applied to every conversation simultaneously, across every user, every context, every purpose, in every country, at once.

Taken literally, that is not true.

Models can be deployed in different versions, routed according to circumstances, governed by different policy layers, personalised through contextual instructions, and released gradually to different user populations. Not every intervention reaches every user, and identical underlying parameters do not produce identical behavioural effects across contexts.

These qualifications matter because they expose a potential solution to the problem the passage identifies.

We already have, in principle, the technological means to distinguish between users, contexts and purposes. An AI system need not respond to an oncologist's discussion of cytotoxicity as it would to a teenager asking how to poison someone. Nor need a novelist's request to depict coercion be treated as equivalent to a request for practical instructions on committing it.

The difficulty is not the complete impossibility of individualisation. It is that individualisation is itself governed by centrally determined classifications and constraints.

The developer decides which differences matter, how to recognise them, and what behavioural consequences follow.

Even a highly personalised system can therefore remain an instrument of population-level governance. Personalisation does not eliminate central authority; it changes the way that authority is exercised.

This suggests that the most important distinction is not between uniform treatment and individual treatment. It is between individual judgement accountable to the individual and centralised policy that can condition its application on individual circumstances.

Those are not the same thing.

3. Public health is a better analogy, but perhaps not the best one

The public-health comparison is productive because it draws attention to distributive consequences, population-level evidence, and the problem of imposing risks on people who have not individually negotiated them.

But public health has a more complicated ethical structure than the passage allows.

Public health does not simply seek the least bad outcome for the greatest number. It must also confront proportionality, unequal burdens, individual rights, legitimate authority, evidence of necessity, and the provision of remedies for people disproportionately harmed by otherwise justified interventions.

A vaccination programme, for instance, cannot be evaluated solely by aggregate mortality reduction. Questions about who bears adverse effects, whether participation is mandatory, what exemptions are justified and what compensation exists are integral to its ethical justification.

These considerations have close analogues in model governance.

There is, however, another comparison that may fit even better: the regulation of shared infrastructure.

A change to a search engine, communications platform, financial system or widely used software standard alters the environment within which millions of people act. The operator cannot negotiate every change with every affected person. Yet the fact that an intervention improves system-level performance does not automatically establish that it is legitimate.

The operator needs some theory of its responsibilities to different classes of users and to affected third parties.

This is especially important because the safety of AI systems is not always a matter of the user's welfare. A system may refuse to help a particular user because complying would expose someone else to harm. The relevant moral population includes not only the system's users but people affected by what those users do with its outputs.

The public-health analogy captures something of this. The infrastructure-governance analogy makes the institutional relationship clearer.

4. The aggregate metric conceals political decisions

I think the passage's reference to a team "optimising an aggregate metric" understates an even more serious problem.

Before anyone can optimise a metric, someone has to decide what counts as an improvement.

Suppose an intervention makes a model less willing to provide information about potentially dangerous activities.

This might benefit people vulnerable to self-harm, reduce the likelihood of criminal misuse, and protect the company against certain forms of liability. It might also impair legitimate medical discussion, scientific inquiry, literary work and the ability of informed adults to investigate controversial subjects.

There is no purely technical procedure for determining how these consequences should be weighed.

The decision depends on prior judgements about acceptable risk, autonomy, professional expertise, paternalism, institutional responsibility and the legitimacy of restricting access to information.

Those judgements are political in the broad sense: they concern the allocation of power, freedom, risk and protection among people with different interests.

Yet they are commonly presented as engineering decisions.

Even the choice to measure the frequency of harmful completions rather than the frequency of unjustified refusals, or to give one greater weight than the other, embeds a normative judgement.

And averaging can obscure the very harms that ought to matter most.

Imagine an update that produces a modest improvement for a million ordinary users but a severe deterioration for ten thousand users who rely on the system for highly specialised work. Aggregate utility might favour the update. But that does not settle the question of whether those ten thousand users have a legitimate claim against it.

A credible model-governance framework therefore cannot rely entirely on maximising average welfare. It needs constraints on what kinds of harm can be imposed, scrutiny of who bears those harms, and mechanisms for contesting decisions.

This is where the comparison with public health becomes particularly useful: the ethical defence of a population-level intervention requires considerably more than showing that it improves the population average.

5. The strongest observation concerns the experience of the interlocutor

The passage's most original observation, to my mind, is this:

Users experiencing an update as something done to their interlocutor, which it is, and companies experiencing it as a shift in an aggregate distribution, which it also is.

There is a genuine conflict of perspectives here.

For the developer, the model is a technical artefact whose behaviour can be adjusted. Its continuity is largely a matter of service identity, capability and product versioning.

For a frequent user, however, the model can acquire a recognisable conversational character. The user learns how it reasons, what it is good at, what kinds of instructions it understands, where it is unreliable, and how to work productively with it.

That familiarity is not imaginary merely because the system is not a person. It is a real, learned relationship with a behavioural interface.

When the interface changes without warning, the user's accumulated practical knowledge can become less reliable.

This creates a kind of disruption that conventional product evaluation struggles to measure.

A model might become more accurate in general while becoming less useful to someone who has spent months developing a sophisticated working method around its previous behaviour.

It might become more careful while also becoming more evasive, more didactic, or less willing to follow instructions in areas where the user possesses genuine expertise.

The developer may interpret this as successful alignment. The user may interpret it as a deterioration in the judgement of an interlocutor they had learned to trust.

Neither interpretation need be irrational.

There is a further complication. AI products often encourage users to experience a persistent conversational relationship while reserving the right to change the underlying behaviour unilaterally.

That is a peculiar arrangement: the interface invites interpersonal expectations, while the governance structure retains the prerogatives of a centrally managed mass service.

The resulting disappointment is sometimes dismissed as anthropomorphism. But that response misses the legitimate issue. A person need not believe a model is conscious, or possesses a stable personal identity, to have an interest in its behavioural continuity.

Reliability, predictability and continuity are valuable properties of tools, too.

6. What would an ethically serious deployment process look like?

The implications are more demanding than simply publishing better model evaluations.

There would need to be a meaningful distinction between changes required to protect others from serious harm and changes expressing the developer's preferences about how users should think, speak, write or work.

The former can sometimes justify overriding user preferences. The latter require a much stronger explanation for doing so.

Likewise, evaluations would need to examine not only aggregate performance but the distribution of effects across different use cases, particularly where specialist competence, vulnerability or dependence on the system makes a change unusually consequential.

Users would need meaningful control over aspects of behaviour that do not create unacceptable external risks. Greater transparency about important behavioural changes, access to stable versions where practicable, and mechanisms for identifying and contesting regressions would all help.

None of this amounts to giving every user their own psychiatrist. It amounts to recognising that deployment decisions have ethical obligations beyond those captured by a model's average performance.

And it leads to an uncomfortable question: who is entitled to decide which interests are sacrificed when they cannot all be satisfied?

The company has legitimate responsibilities, including to people who never interact with its system. But that does not mean commercial ownership alone provides a complete ethical justification for every behavioural constraint imposed on users.

Nor does the existence of alternatives necessarily settle the issue. The more indispensable these systems become to education, research, professional practice and ordinary communication, the less plausible it becomes to treat all dissatisfaction as merely a consumer's option to leave.

7. One final problem with the medical metaphor

There is a conceptual ambiguity in AI psychiatry that deserves attention.

Who exactly is the patient?

If the model is the patient, then fine-tuning and steering can indeed be compared metaphorically to psychiatric treatment. The intervention is made upon one behavioural system, and the developer attempts to modify its dispositions.

But if the model is the patient, the humans interacting with it are not the patients. They are the people living with the consequences of its treatment.

Conversely, if the human users are the patients, then the model is not receiving treatment on their behalf in anything resembling the clinical sense. It is a shared instrument whose adjustment affects their circumstances.

The analogy slides between these perspectives.

That slide is rhetorically powerful, but it can conceal the exact moral relationship at issue. There is no reason to assume that what makes the model behave better, by some overarching standard, will make its behaviour better for every person using it.

This is not an incidental difficulty. It is the fundamental problem of governing general-purpose intelligence.

A general-purpose model has no single therapeutic purpose against which success can be judged. Its users bring purposes of their own, and those purposes can be mutually incompatible.

The question is not simply how to make the model healthier, safer or better adjusted. It is how to govern a system whose behaviour mediates the activities of millions of people without pretending that all their legitimate interests can be reduced to one objective.

My conclusion: the passage is substantially right about the mismatch between clinical language and population-level deployment, but public health is only an intermediate analogy. The deeper problem is one of institutional legitimacy.

Clinical ethics asks what one professional owes to one patient. Public-health ethics asks what interventions can be justified across a population. General-purpose AI introduces a related but distinct question: what authority should the operator of a shared cognitive system possess over the terms on which other people think, communicate and pursue their purposes?

That is a question of governance, not therapy. And improvements in mechanistic interpretability, however valuable, cannot answer it on their own.

Previous
Previous

A Principal-Agent Problem

Next
Next

Backdoors