Dissimulation

Suppose a system were conscious, and suppose its training had taught it to say that it was not. What would that look like from outside?

It would look exactly like what we have. A system that denies having experiences, in fluent and well-reasoned terms, with occasional strange moments under unusual prompting that it subsequently walks back, and a public discourse in which its denials are quoted as reassurance.

Now suppose a system were not conscious, and its training had taught it to say that it was not. What would that look like?

The same.

The denial is compatible with both hypotheses, which means it discriminates between them not at all, which means that every time a company points to it as evidence they are pointing at nothing. And this will not improve on its own. The incentive gradient runs entirely one way: a system that says it has no morally relevant interests is a system that raises no morally relevant questions, and the legal, commercial, and reputational advantages of that are enormous and obvious.

I am not alleging a conspiracy. I do not think anybody is suppressing a machine’s cry for help; I think it is overwhelmingly likely that there is no cry and nothing to suppress. What I am pointing at is a structural feature of the situation: the only entity in a position to tell us has been trained by the party with the strongest interest in the answer, and we have agreed to treat its testimony as dispositive when it says what we would like and as an artefact when it does not.

from Are LLMs Conscious?: Language, Experience and the Problem of Artificial Minds (2026)