On Testimony

When you tell me you are in pain, a causal chain connects your pain to your words. Tissue damage, nociception, cortical processing, some further process by which the state becomes available to your linguistic system, and then the sentence. The chain can be interfered with — anaesthesia severs it, aphasia severs it, and there are disorders in which it is severed in remarkable and specific ways — but in the normal case it is there, and its presence is what makes your report about your pain rather than merely coincident with it.

Now consider a language model producing the sentence ‘I find this distressing.’ What caused those words?

The honest answer is: the training corpus, refracted through the optimisation process and the current context. Those particular words are likely here because, in the enormous record of human writing, that construction follows contexts of this shape. If we could somehow establish that the model was in an internal state with a distressing character — grant it, for the sake of argument — that state would not be what produced the sentence. The sentence would have been produced anyway. And if the state had been absent, or reversed, or replaced with something with no human analogue at all, the sentence would still very probably have been the same, because the sentence is determined by the statistics of what humans say in circumstances like these.

This is the crux, and it is not a point about what is going on inside. It is a point about counterfactual dependence. A report is evidence of a state only if it would have been different had the state been different. The model’s reports about its inner life do not have this property, or have it only in ways we cannot currently separate from the ways they lack it. Which means that the single richest source of evidence about minds — testimony — has been rendered, not inaccurate exactly, but disconnected.

from Are LLMs Conscious?: Language, Experience and the Problem of Artificial Minds (2026)