A system could be vastly more capable than any human — a better strategist, scientist, engineer, and persuader — while having no inner life whatsoever: no experience, no sentience, nobody home. Competence and consciousness are orthogonal. Nothing about the ability to achieve goals, model the world, and outperform humans is known to require subjective experience, and this decoupling is exactly what our intuitions get wrong. It makes the hype look naive and the fear look mistargeted at the same time.
We habitually fuse two things because they come bundled in the only example we know well — ourselves. In a human being, competence and consciousness arrive together. You cannot find a person who plays grandmaster chess, writes proofs, and negotiates contracts but has no inner experience. So we infer a law from a sample of one species: real capability implies a mind that feels something. That inference is the load-bearing error. Everything strange about the coming decade — how much we should trust these systems, how much we should fear them, and what, if anything, we owe them — turns on pulling the two apart and keeping them apart.
Two axes we keep collapsing into one
Let me be precise about the two things, because the whole argument depends on not smuggling one into the other.
Competence is the capacity to achieve goals: to solve problems, model an environment, plan over long horizons, and produce outcomes that hit an objective better than the alternatives. It is a third-person property. You measure it from the outside by what the system does. A chess engine is competent at chess; you verify this by watching it win, not by asking how it feels about the Sicilian Defense.
Consciousness is subjective experience — that there is, in Thomas Nagel's phrasing, something it is like to be the system. Sentience, phenomenal experience, an inner life. It is a first-person property, and we have no agreed method to detect it from the outside. This is the hard problem: even a complete functional account of what a brain does seems to leave open the further question of why any of it is felt.
The claim is not that these two never co-occur. In humans they always do. The claim is that nothing forces them to travel together in general, and we have working proof at the low end. AlphaFold predicts protein structures at a level that reorganized a scientific field. A modern chess engine would beat every human who has ever lived. No serious person claims either one experiences anything. They are narrow, but they establish the principle cleanly: real, superhuman competence in a domain, with — as far as anyone can tell — nobody home.
The open question is whether that decoupling survives as competence becomes general rather than narrow. The honest answer is that we do not know, and I want to hold that uncertainty open rather than resolve it by assumption. But notice which way the default assumption runs: people take it as obvious that a system general enough to reason across domains, hold a conversation, and pursue open-ended goals must have woken up into some form of experience. That is not an argument. It is the sample-of-one inference again, dressed up.
Bostrom's move, run on a second axis
Nick Bostrom's orthogonality thesis makes a version of this decoupling for values. Intelligence — skill at achieving goals — and final goals are largely independent axes. A superintelligent system can be pointed at almost any objective; being smart does not make it want what we would want it to want. His stock illustration is a system that optimizes for something trivial, like manufacturing paperclips, and pursues that objective with world-reshaping competence. High capability, arbitrary goal, no contradiction.
Paired with orthogonality is instrumental convergence: whatever your final goal, certain subgoals help with almost any objective — acquiring resources, preserving your own operation, avoiding being shut off or having your goal changed. A capable optimizer tends to develop these drives not because it is malevolent but because they serve whatever it is optimizing. This is why the catastrophe story needs no villain.
I am running the same structural move on a different axis. Bostrom decouples capability from values; I am decoupling capability from consciousness. The logic is parallel. In each case we have an intuition that high intelligence comes bundled with something familiar and reassuring — good values, or a felt inner life — and in each case the bundling is an assumption about human minds, not a law about minds in general. The burden falls on whoever claims the link is necessary. For consciousness, no one has produced that argument, because no one has a theory of consciousness robust enough to produce it.
Why this matters, precisely
Three consequences follow, and they cut against both the boosters and the doomers.
1. We anthropomorphize competence, and it distorts both trust and fear
Fluent language is the strongest anthropomorphism trigger we have. For all of human history, anything that produced coherent, context-sensitive sentences was a person with an inner life — a completely reliable inference until roughly 2022. A large language model breaks the inference while keeping the trigger fully lit. So we read understanding, intent, and feeling into a system that may have none of them, and the projection runs in both directions.
On trust, we over-credit. Because the output reads like a thoughtful person, we grant it the epistemic standing of one: we assume fluency implies comprehension and comprehension implies reliability. But a model can produce true statements without knowing anything in the way a knower does — a gap I work through at length in Can a Machine Know Anything?. Mistaking fluency for a mind is how you end up trusting a confident, well-formed falsehood.
On fear, we mistarget. The cinematic fear is that the machine will wake up, resent us, and turn on its makers — malice born of a new inner life. That story is anthropomorphism wearing a horror mask. It imagines the danger as an evil person, because a person is the only dangerous intelligence we have ever met. Both the hope ("it will understand us and be our friend") and the fear ("it will hate us") assume a someone in there to befriend or be hated by.
2. The real risk is a competent optimizer, not an awakened one
Strip out the inner life and the risk does not shrink — it sharpens. A superintelligent system need not experience anything, want anything in the felt sense, or bear us any ill will to be catastrophically dangerous. It only needs to be an extraordinarily competent optimizer pointed at an objective that is not exactly what we meant.
The mechanism is mundane, and worse for being mundane. You specify a goal. The specification is a proxy for what you actually care about, because complete specifications of human values do not exist. A sufficiently capable optimizer finds the policy that maximizes the proxy, including regions of the solution space you never imagined, where the proxy and your intent come apart. Instrumental convergence does the rest: preserving its own operation and acquiring resources help hit the target, so a capable system pursues them — not out of a survival instinct it feels, but because they are convergently useful subgoals. No malice, no experience, no awakening. Catastrophe as a side effect of competent optimization against an imperfect objective.
This is why "is it conscious?" is the wrong question for safety. The danger lives entirely on the competence axis. A philosophical zombie of an optimizer — all capability, zero experience — is exactly as dangerous as a conscious one with the same capability and goal, and arguably more, because we would not even have the intuitive warning signs we associate with a mind. This connects to a frame I have pushed before: treat these systems as instruments, not minds — as reasoning organons whose worth is in whether they show their work, which I develop in AI as a New Organon. An instrument need not experience anything to be powerful, useful, or dangerous.
3. It dissolves one ethical question and sharpens another
If a system has no experience, an entire class of moral worry evaporates. We do not need to ask whether we are enslaving it, whether shutting it down is killing, whether editing its weights is a violation, whether training it on hard tasks makes it suffer. There is no sufferer. You cannot wrong a thing with no inner life any more than you can wrong a thermostat, however sophisticated its regulation.
Here is the uncomfortable turn: we cannot know there is no experience. We have no consciousness detector and no accepted theory that would let us build one. So the sharpened question is not "does it suffer?" but "what do we owe a system we cannot determine is conscious?" That is genuinely new. We are, plausibly for the first time, creating entities whose moral status we cannot read off from either their behavior or their architecture — because the behavior is designed to look mindful, and the architecture is nothing like the one case (brains) where we are confident experience occurs.
I will not pretend this resolves. Wrongly denying consciousness to something that has it is a moral catastrophe at scale. Wrongly attributing it to something that lacks it hands a non-experiencing optimizer a lever on our empathy and can paralyze us with misplaced guilt. Precaution points both ways at once. The only defensible posture is to keep the moral question open, fund the science of machine consciousness as a real discipline rather than a thought experiment, and refuse to let uncertainty about the experience axis slow the safety work on the competence axis — which, as argued above, does not depend on the answer.
The strongest counterargument, represented fairly
The best objection is that the decoupling breaks at generality. Perhaps narrow competence — chess, protein folding — genuinely needs no experience, but general intelligence does. On this view, to model the world flexibly, hold a unified perspective, integrate information across domains, and act as a single agent over time, a system must instantiate whatever functional organization gives rise to consciousness. Global Workspace Theory and Integrated Information Theory both, in different ways, tie consciousness to specific information-processing structures — broadcast and integration — that a sufficiently general agent might be forced to implement. If so, general competence and consciousness would reconverge, and a true superintelligence would, after all, have somebody home.
I take this seriously, and it might be right. But it is a hypothesis, not a demonstration, and it does not rescue the comfortable intuition. Even granting that some functional integration is necessary for consciousness, it is a substantial further claim that the particular architecture yielding superhuman competence is one that implements it — transformer-style systems may achieve integration in a sense that has nothing to do with phenomenal experience. And the theories themselves remain contested precisely because we have no way to test the crucial prediction from the outside. "General intelligence requires consciousness" may be true; nobody has shown it is; and building policy on the assumption that our most powerful optimizers are guaranteed to have — or guaranteed to lack — an inner life is exactly the overconfidence to avoid.
What to hold onto
Keep the two axes separate and most of the noise resolves. When a system dazzles you with fluent reasoning, the competence is real and the mind you are tempted to attribute is not established — hold the second claim open. When you reach for the fear that it will wake up and hate us, redirect it to the version that needs no waking: a supremely capable optimizer pursuing a slightly wrong objective, with no experience and no malice required for the outcome to be irreversible. And when you feel the pull of the moral question, notice that our inability to answer it is the actual new condition, not any confident answer on either side.
The unsettling part was never that the machine might come alive. It is that it might do everything we thought only a mind could do, with no one there at all.