The central obstacle to understanding what a neural network actually does is not that it is large, or that its math is exotic. It is that the network stores more concepts than it has neurons, and it does this by letting concepts share. A feature is not a neuron; it is a direction in activation space, and the network packs many more directions than it has dimensions by allowing them to overlap slightly. The consequence is direct and unforgiving: individual neurons are polysemantic — one neuron fires for many unrelated things — so you cannot read a model by reading its neurons. This is called superposition, and it is why "we built it but can't fully read it" is a literal statement about frontier systems, not a marketing line.
The problem is precise, the arithmetic that forces it is short, and the leading fix — sparse autoencoders — is real but partial. What follows is each of those, and then what it actually licenses you to believe about a model you cannot fully read.
Features are directions, not neurons
Start with the working hypothesis behind most of modern interpretability: the linear representation hypothesis. It says a network represents human-meaningful features as directions in its activation space. "This text is in French," "this token is inside a URL," "the sentiment here is negative" — each corresponds, roughly, to a vector, and the presence and strength of the feature is how far the activation points along that vector. Concepts add and subtract in a way that made the old word2vec analogies work, and it is the reason interpretability researchers talk about directions at all.
If features were neurons — if each concept lived on its own dedicated unit — interpretability would be tedious but tractable. You would probe neuron 4,113, find it fires for "legal contracts," and move on. Early work on vision models found a few such clean units: curve detectors, high-low frequency detectors. It looked like the network might be a readable dictionary where each entry is a neuron.
It is not. When researchers looked hard, most neurons turned out to fire for grab-bags of unrelated inputs — a classic example being a single vision unit that responds to cats, cars, and cat-shaped cars, but the everyday version is a language neuron that lights up for academic citations, HTTP requests, and Korean text with no coherent story linking them. That is polysemanticity, and it is not noise or a training defect. It is the fingerprint of a network doing something clever with limited room.
The arithmetic that forces sharing
Here is why the network has no choice. A middle layer of a transformer might have on the order of ten thousand dimensions. The number of features it would be useful to represent — named entities, syntactic roles, topics, tones, factual associations, code patterns — is vastly larger than ten thousand. You cannot assign each feature its own orthogonal axis, because you run out of axes. So the network does the only thing available: it stores features as directions that are not orthogonal, that overlap a little, and it tolerates the interference.
The reason this works is a fact about high-dimensional geometry that is genuinely counterintuitive. In a space of dimension d, you can fit only d mutually perpendicular vectors. But relax "perpendicular" to "almost perpendicular" — allow a small angle away from 90 degrees — and the number of directions you can pack grows exponentially with d, not linearly. This is the neighborhood of the Johnson–Lindenstrauss lemma: high-dimensional spaces have room for enormously many nearly-orthogonal directions.
Make it concrete with the interference itself. Take two random unit vectors in d dimensions. Their dot product — the interference between them, how much one bleeds into the other when you try to read it — has mean zero and a standard deviation of about 1/√d. For a width of d = 12,288 (the residual stream of GPT-3), that is roughly 0.009. So a feature you read off along its own direction comes back with a signal of 1, contaminated by cross-talk from every other active feature at a level of about nine parts in a thousand each. If only a handful of features are active on any given token — and real features are sparse, most of them off most of the time — those small contaminations sum to something the network can subtract off or simply ride through. The network is running an error-correcting code it discovered by gradient descent: pack aggressively, keep the collisions rare and small, decode approximately.
This was not inferred from vibes. Anthropic's Toy Models of Superposition (Elhage et al., 2022) built deliberately tiny models where the true features were known by construction, then watched the models choose to represent more features than they had dimensions exactly when the features were sparse. The toy models even arranged their overlapping features into clean geometric structures — antipodal pairs, triangles, pentagons, tetrahedra — trading representation fidelity against interference as a function of how sparse the features were. Superposition is not a hypothesis in the toy regime; it is a reproducible phenomenon with a geometry.
The load-bearing conclusion: a neuron is the wrong unit of analysis. The neuron is just one coordinate axis of the activation space, and the features are directions that cut across many axes at once. Asking what a neuron means is like asking what the number 0.31 in the third slot of a vector means — the meaning lives in the whole vector's alignment with feature directions, not in any coordinate. Reading neurons to understand a network is a category error, and it is the reason the intuitive picture of interpretability keeps failing.
The leading response, without the hype
If features are directions hidden in superposition, the natural move is to recover those directions — to take the tangled activation vector and decompose it back into the sparse set of underlying features that produced it. That is the job of a sparse autoencoder (SAE), and it is the technique carrying most of the current progress.
The construction is clean. You train a small network with one hidden layer that is wider than the activation you feed it — an overcomplete dictionary, maybe eight or thirty times as many hidden units as the layer has dimensions — and you force that hidden layer to be sparse, penalizing it (an L1 penalty on the activations, in the standard recipe) so that only a few of its units may fire to reconstruct any given input. The bet is that the sparse dictionary the SAE learns will line up with the network's true, superposed features: each dictionary element a single direction, each direction one human-meaningful concept. You are trading the network's compressed, entangled code for a longer but disentangled one.
It works better than it had any right to. Towards Monosemanticity (Anthropic, 2023) took a one-layer transformer and pulled out features that were dramatically more monosemantic than the raw neurons — features for DNA sequences, for legal language, for specific programming patterns, each firing for one coherent thing. Scaling Monosemanticity (2024) pushed the method onto a production-scale model, Claude 3 Sonnet, and recovered millions of features, including abstract and multilingual ones and a now-famous "Golden Gate Bridge" feature that, clamped on, made the model bring every conversation back to the bridge. That intervention is the real evidence: you found a direction, you pushed on it, and behavior moved the way the interpretation predicted. That is closer to justified belief about an internal than anything neuron-reading ever delivered.
Now the honesty the field itself insists on. The reconstruction is lossy — the SAE never perfectly rebuilds the activation, and the leftover variance is representation you have simply failed to decode. Features split: change the dictionary size and a single feature can fracture into several finer ones, so the "true" number of features is not well defined and the decomposition is not unique. Many dictionary units come out dead or uninterpretable. And most importantly, naming features is not the same as explaining computation. Knowing that a model has a feature for "unsafe code" and a feature for "the user is asking about chemistry" does not tell you the circuit — how those features are read, combined, and turned into the next token. Tracing a behavior end to end through the features that cause it remains far harder and is nowhere close to complete for a frontier model. The state of the art is real, and it is partial. We can now name a great deal of what a large model represents. We cannot yet audit how it computes.
What this licenses you to believe
The philosophical shape of the problem is worth naming, because it tells you where to put your trust. What we want from interpretability is justification — not merely a true belief that a model is safe or capable, but a belief we hold for the right reasons, grounded in the mechanism. That distinction, between a belief that happens to be true and one that is genuinely warranted, is the old lesson of the Gettier problem: true belief without the right kind of grounding is not knowledge. Right now, our confidence in what a model represents is mostly grounded in behavior — we prompt it, we watch what comes out, we generalize. That is trust by testimony: the epistemic position of believing a witness because they have been reliable before, not because you can check their reasoning. It can be well-calibrated, and it is still not inspection.
This is why superposition is not merely an academic puzzle. If you cannot reliably read what a model represents, you cannot fully verify it. You cannot certify that a dangerous capability is truly absent rather than dormant in some direction you never decoded, and you cannot prove that a well-behaved model is well-behaved for the reasons you hope rather than for reasons that will not survive a new distribution. The safety and capability claims we make about these systems rest on the outside of the model — its outputs — because the inside is only partly legible. It is the same wall that makes confidently wrong outputs so hard to catch: the failure I describe in Hallucination Is a Training-Objective Problem is invisible from the outside precisely because we cannot read the internal state that would reveal the model has no grounding for what it just asserted. And it is why you can't evaluate an agent you can't specify — trajectory-level behavior is hard to score for the same reason internals are hard to read: the ground truth we want lives in a representation we can only decode in part.
So calibrate. Interpretability today can find and name individual features. It can sometimes intervene on a feature and predict the behavioral change, which is genuine mechanistic evidence. It cannot yet give you a complete, circuit-level account of why a frontier model produced a specific output, cannot enumerate everything the model represents, and cannot guarantee the absence of a capability. Treat a clean SAE feature as strong evidence about one direction, not a warranty on the whole system. When someone claims a model is safe "because we understand it," ask which sense of understand — a named feature, or a traced circuit — and price the difference.
We built machines that hold more ideas than they have parts to hold them, by letting the ideas overlap. That is the same move that makes them powerful and the same move that makes them opaque. The reading is getting better. The book is still mostly closed.