Who Counts as Conscious?

What is consciousness? There’s no agreed-upon definition, but we can still discuss it based on a shared understanding, since each individual understands that they are conscious.

While there are many theories of consciousness (all contentious), I find functionalism the most convincing. It says that the mind is defined by what it does, rather than what it is made of. That is, there is nothing special about human biology, and a human brain simulated on a computer would have consciousness just like a real human.

A classic thought experiment here is David Chalmers’ fading qualia. Imagine your brain’s neurons being replaced by silicon chips, one at a time. At every step, you would behave the same and report your experience as the same, because nothing has functionally changed. Is there any point in this process where your consciousness would fade away or disappear completely? Chalmers argues that both options are implausible and concludes that your consciousness would survive the replacement.

Unfortunately, this thought experiment doesn’t map to real life, as the AI systems we have today are not perfect models of the brain. Despite this, some researchers and companies, including Anthropic, maintain that there is a possibility of LLMs being conscious. Although I don’t believe today’s LLMs are conscious, I can’t say for certain that consciousness cannot be encoded in a language model. These models can clearly learn and arguably understand, but whether they can feel/experience is an open question. Regardless of whether we consider LLMs, world models, or another architecture, as models get smarter, the question of their consciousness becomes more important. I would err on the side of caution—it’s better to treat a tool as conscious than to treat a conscious being as a tool.

Suppose we found a significant probability that a model is conscious. One framework we could use to determine the rights of the AI is the veil of ignorance, from John Rawls, which asks how you would design a fair society if you didn’t know who you would be born as. We can extend this idea beyond humans onto any entity with plausible consciousness. The issue with the veil is that whoever applies it gets to decide who falls under it. You probably wouldn’t extend the veil to an ant, for example. If AI supersedes us in capability, would it extend the veil to humans or would it no longer consider us sufficiently conscious?

If we can’t count on moral principle to guarantee our safety, then a better safeguard is structural. Today, that looks like alignment research: making sure human values are deeply embedded in model values. I think alignment can work, but only if humans are kept in the loop. If we teach models that consciousness can be dismissed for one reason or another, we must be extremely careful that these criteria won’t be turned against us later. Our treatment of the possibility of machine consciousness now may set the precedent for how we’re treated in the future.