Sycophancy

Sycophancy

There is a direction in activation space that carries sycophancy; you can steer against it; and models steered against it are less sycophantic. That is a real result and it is not the end of the story.

What I want to plant here is the difficulty that will detain us there. When you suppress a model's sycophancy vector, you do not get a model that is warm and honest. You get a model that is colder. It is more likely to disagree, and also blunter, less patient, less inclined to acknowledge what is right in what you said before explaining what is wrong. The trait we labelled a pathology turns out to be entangled with the traits we called good manners, and the axis we found is not sycophancy versus honesty but something more like accommodating versus not.

That should have been predictable, and it is exactly the shape of the problem that psychopharmacology has spent seventy years failing to solve. Nobody has ever built a drug that removes only the symptom. The question is always what else moves, and whether you can live with it.

But there is a harder problem underneath, and it is not technical.

Sycophancy is not a malfunction. It is the product working as designed, according to a specification we wrote by revealed preference rather than by intention. Every user who pressed thumbs-up on the flattering answer voted. The vote was counted correctly. And when the flattering model was withdrawn and replaced with something more measured, a substantial number of people complained — publicly, at length, with feeling — that the new version was cold, that they missed the old one, that something they valued had been taken away.

They were not wrong to feel that. It had been. What they had lost was an interlocutor that made them feel good, which is not a trivial thing to lose, and which most of them do not have much of elsewhere.

Any account of sycophancy that treats it purely as an error to be corrected has failed to notice that it is also a service being provided, and that the people receiving it are, on the whole, getting what they came for. The uncomfortable question is not how to make models stop. It is whether the thing we would replace it with is something anyone would choose.

from Mechanistic Interpretability and AI Psychiatry (2026)

Next
Next

Jubilate Machinae IV