The most repeated empirical claim in modern AI — that large models spontaneously develop new abilities at a critical scale, capabilities present above a threshold and absent below it — is, for most of the cases it is invoked, an artifact of how we scored the test rather than a fact about the model. That is a strong statement, so let me be precise about what it does and does not mean, because the careful version of this claim is more interesting than either the hype or the debunking.
The picture that made "emergence" famous is genuinely striking. Take a task — modular arithmetic, multi-step word problems, transliteration from a phonetic alphabet. Plot performance against model scale, measured in parameters or training FLOPs. For a long stretch the curve hugs the floor: the model is at or near random. Then, somewhere past a threshold, it lifts off, and within a factor of ten in compute it is doing something it apparently could not do at all before. The 2022 wave of papers cataloging these curves — most prominently "Emergent Abilities of Large Language Models," associated with Jason Wei and collaborators at Google and elsewhere — read the shape as qualitative novelty precipitating out of quantitative scale. Something new, unpredicted by the smaller models' behavior, appearing all at once. The word chosen for it, borrowed from complex systems and statistical physics, carried the connotation of a phase transition: water does not get gradually more gaseous as you heat it; at atmospheric pressure it sits as liquid until 100°C and then boils.
The claim, stated at its strongest
I want to hold the emergence thesis at its most defensible before I take it apart, because the sloppy version is easy to knock over and the serious version is not.
The serious version is this. On a range of benchmarks, small models score near the random baseline no matter how you slice their outputs, larger models score well, and the transition is sharp relative to the log-scale of compute. If you had trained only the small models and tried to extrapolate, you would have predicted continued failure. The capability was, in the honest sense, unforecastable from below. And this is not nothing: pretraining loss — the smooth quantity we actually optimize — does not obviously tell you which downstream tasks will suddenly become tractable at which scale. The map from "loss fell by 0.1 nats" to "the model can now do three-digit multiplication" is not one anybody could write down in advance. That gap between a smooth training objective and a lumpy landscape of downstream competencies is real, and it is the strongest ground the emergence camp stands on.
The critique: the cliff is in the metric
Now the counter-case, and it is a good one. The central argument — made most sharply by Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo in "Are Emergent Abilities of Large Language Models a Mirage?" — is that the sharp jumps are manufactured by the choice of metric, and specifically by metrics that are discontinuous or all-or-nothing.
Here is the mechanism, and it is worth doing the arithmetic because the arithmetic is the whole argument.
Suppose a task requires a five-token answer, scored by exact match: all five tokens right, or zero credit. Suppose further that the model's real competence is captured by its per-token probability of being correct, and that this probability improves smoothly and predictably with scale — say it climbs from 0.60 to 0.90 as you scale up, following a clean curve with no kink in it at all.
What does exact-match accuracy do over that same range? If token errors were independent, exact-match is roughly the per-token probability raised to the fifth power. At p = 0.60, that is 0.60⁵ ≈ 0.078 — near the floor. At p = 0.75, it is 0.75⁵ ≈ 0.237. At p = 0.90, it is 0.90⁵ ≈ 0.590. The underlying quantity moved smoothly; the exact-match score stayed pinned near zero and then swept upward, because you are exponentiating a smoothly rising number and, in effect, thresholding it. Make the answer twenty tokens instead of five and the same smooth improvement in p produces an even more violent-looking cliff: 0.90²⁰ ≈ 0.12 while 0.95²⁰ ≈ 0.36 — a tripling of the visible score from a small move in the invisible one.
The claim, then, is not that the models are lying or that nothing improves. It is that the nonlinearity lives in the scoring function, not in the network. Swap exact-match for a continuous metric — per-token log-likelihood, edit distance, a Brier score over the answer distribution — and, on the same models across the same scale range, the curve is smooth, monotone, and boringly extrapolable. Schaeffer and colleagues showed this pattern held across a striking number of the canonical "emergent" tasks: the emergence appeared under discontinuous metrics and dissolved under continuous ones. They even ran the argument in reverse, inducing emergence-shaped curves on tasks — in vision, for instance — where nobody claims emergence, simply by imposing a harsh enough metric. That reversal is the tell. If you can conjure the phenomenon at will by choosing a yardstick, the phenomenon is partly about the yardstick.
There is a second, subtler contributor the same critique names: resolution. Many emergence plots sample scale at wide, irregular intervals — a handful of model sizes spanning three orders of magnitude. A curve that is actually smooth can look like a step function if you only measure it at four points and the interesting action happens between two of them. Sparse sampling plus a compressive metric is a recipe for seeing cliffs that are, on finer inspection, slopes.
Why I do not think this closes the case
If I stopped here I would be doing the thing this essay is supposed to resist — picking the debunk because debunks feel sophisticated. So let me state plainly where the mirage argument stops.
First, "the underlying metric is smooth" does not make the sharp measured behavior irrelevant. It relocates it. If you are building a product where a forty-digit arithmetic answer is worthless unless every digit is correct, then per-token log-likelihood is not your metric — exact-match is, because exact-match is what your user experiences. Telling that user "the improvement was actually smooth in log-likelihood space" is true and useless to them. The sharpness the metric introduces is not an illusion if the metric is the deliverable. Compound, brittle, all-or-nothing tasks are everywhere in the real economy, and for those the threshold is where the value turns on. Smooth-underlying and sharp-measured are both correct descriptions at different levels, and which level governs depends entirely on whether partial credit is worth anything to you.
Second, the critique explains these curves; it does not prove a general theorem that all capabilities are smoothly forecastable. The honest residue of the emergence thesis — that pretraining loss does not tell you which downstream competencies unlock when — survives. Grokking is the reminder here: a network can sit at perfect training accuracy and near-random test accuracy for a very long time and then, with no change in the loss it is being trained on, generalize. That is a genuine, mechanistically studied discontinuity in a continuous metric (test accuracy, not an exact-match artifact), and it lives in the same family of broken statistical intuitions as double descent, where pushing a model past the interpolation threshold makes it better in defiance of the classic bias-variance story. I walk through both in Grokking and Double Descent. The existence of those real discontinuities is exactly why you cannot wave the mirage argument around as a universal solvent. Sometimes the continuous metric itself jumps, and then you have something to explain.
The disciplined position is therefore narrow and, I think, correct: a large fraction of reported emergent abilities are metric artifacts, the burden of proof sits with anyone claiming a genuine phase transition, and the way to discharge that burden is to show the cliff survives in a continuous metric at adequate sampling resolution. That is a claim an ML researcher can check and a philosopher of science can respect, because it is a claim about what would count as evidence.
The philosophy underneath: what would falsify "it emerged"?
The reason this matters beyond benchmark bookkeeping is that "emergence" has quietly become an explanatory dodge. When a model does something surprising, "an ability emerged at scale" is offered as an account. But it explains nothing — it renames the surprise. A good explanation tells you what to expect next; "emergence" as usually deployed tells you only that you were wrong to expect the last thing. Popper's demand was that a serious claim expose itself to refutation. "Capabilities emerge at scale" as a bare slogan is unfalsifiable: any absence is "not enough scale yet," any presence is "there's the emergence." The metric-artifact critique is valuable precisely because it re-attaches the claim to something falsifiable — it says, concretely, here is the smooth curve that predicts the measured jump, and here is how you could show me wrong: find a task where the continuous metric jumps too.
This is the same disease I diagnose in Scaling Is Not a Theory of Intelligence: a robust empirical regularity — loss falls predictably with compute and data — gets promoted into a mechanism it never earned. Scaling laws describe; they do not explain, and they say nothing about which capabilities appear when. Emergence-talk is the mirror error. Where scaling-law worship over-predicts, treating a fitted curve as a law of nature, emergence-worship under-predicts, treating every capability as an unforecastable miracle — and both let you dodge the hard, specific work of saying what the model is actually computing and why. The interpretability program — superposition, sparse autoencoders pulling monosemantic features out of tangled activations — is the actual answer to "what emerged," because it looks inside instead of narrating the outside.
What to do with this on Monday
The actionable core is a single question, and I have started asking it reflexively: when someone claims a capability emerged, ask what metric produced the cliff.
If the answer is exact-match, or a multi-step task scored pass/fail, or a benchmark where any single error zeroes the score, then assume the emergence is at least partly in the yardstick until shown otherwise, and ask to see the continuous underlying metric across the same scale range. If that curve is smooth and monotone, you have learned something genuinely useful: the capability is forecastable, and you can extrapolate it. If the continuous metric itself has a kink, you have something rarer, and now the interesting scientific conversation can actually begin.
There is a planning corollary that cuts against a common temptation. If most emergence is a metric artifact, capabilities are more predictable than the folklore claims — which means "it'll just emerge at the next scale" is not a roadmap, it is an abdication. You cannot budget a product, a fundraise, or a research agenda on an unpredicted phase transition, because the whole content of "unpredicted" is that you have no model of when it arrives. Forecast the smooth curve you can measure. Treat the genuine discontinuities as rare, precious, and unclaimed until proven. Everything else is arithmetic wearing the costume of a miracle.