Skip to content

The Intelligence Explosion Assumes Intelligence Is the Bottleneck. It Usually Isn't.

The intelligence-explosion story hides one premise: that intelligence is the binding constraint on producing more intelligence. It usually isn't. The recursion runs through compute, energy, data, and experiments that don't speed up because the model got smarter.

By Mehdi10 min read
Share
On this page

The intelligence-explosion argument has a hidden premise, and once you see it the whole edifice turns from a deduction into a bet. The premise is that intelligence is the binding constraint on producing more intelligence — that a system smart enough could bootstrap itself to superintelligence on a timescale set by its own cleverness. It usually isn't the binding constraint. Recursive self-improvement runs through compute, energy, data, and experiments that touch the physical world, and none of those get faster because the model got smarter. The loop grows at the pace of its slowest external input, not at the pace of its intelligence, and that single fact is the most under-discussed reason the explosion may be bounded.

I want to make this argument precisely, because it is easy to make badly. The lazy version says "AI is overhyped, physics is hard, relax." That is not an argument, it is a mood. The rigorous version is a claim about the structure of recursions: a feedback loop accelerates without bound only when the thing it produces is also the thing that limits its own production. Break that identity — insert any fixed real-world resource that does not scale with intelligence — and the runaway becomes a climb with a ceiling. The interesting question is not whether there is a ceiling. It is which external input sets its height.

The hidden premise, stated as an equation

Start with what the classic argument actually says. I.J. Good put it cleanly in 1965, in "Speculations Concerning the First Ultraintelligent Machine": define an ultraintelligent machine as one that can far surpass all the intellectual activities of any man, including the design of machines. Then, he wrote, "there would unquestionably be an 'intelligence explosion,'" because the machine could design an even better machine, which could design a better one still — "the last invention that man need ever make." Vernor Vinge dramatized it as the singularity in 1993; Ray Kurzweil turned it into a law of accelerating returns; Nick Bostrom gave it its most careful treatment in Superintelligence (2014), where he framed the rate of intelligence gain as optimization power divided by recalcitrance — how hard you push over how much the system resists.

Bostrom's framing is the honest one, and it already contains the whole objection, though it is usually read past. The fast-takeoff story is not "optimization power rises." It is "optimization power rises while recalcitrance stays low or constant." That second clause is the load-bearing assumption, and it is precisely the assumption the physical world refuses to grant.

Make it concrete with a piece of arithmetic that has nothing to do with AI and everything to do with why the explosion is bounded. Any round of self-improvement is a cycle with parts. Some fraction of each cycle's wall-clock time is spent on things that get faster as the system gets smarter — reasoning, code, design search. Call that fraction accelerable. The rest is spent on things that don't: waiting for a chip to be fabricated, a training run to finish on the compute you actually have, an experiment to return a result from reality. This is Amdahl's law, one of the oldest results in parallel computing, pointed at recursive self-improvement. If a fraction f of the cycle is fixed by the external world, then even infinite thinking speed caps the total speedup at 1/f. If experiments and fabrication are 30% of the loop, the smartest possible mind makes the loop at most about 3.3 times faster. If they are half, at most twice. Not twice per generation compounding forever — twice, full stop, as the intelligence-limited part collapses toward zero and the external part sits there unmoved.

That is the skeptical argument in one line. The explosion is a geometric series only over the accelerable fraction. The fixed fraction is a floor, and the floor is set by the physical world, not by the model.

The three recursions that actually matter, and what gates each

The abstract point lands only when you name the concrete loops people have in mind when they picture takeoff. There are three, and each is throttled by a different external resource.

The AI that designs better chips. This is the purest version of the dream: an AI that improves the hardware it runs on, closing the loop between intelligence and its own substrate. Suppose it works — suppose a system produces a genuinely superhuman chip design in an afternoon. It then has a file. To turn that file into compute it needs a leading-edge fab, and there are a handful on Earth, each costing on the order of tens of billions of dollars and taking on the order of three to five years to build, dependent on a single supplier of EUV lithography machines, ultrapure materials, and a workforce of specialists. The design step just got infinitely fast. The realization step is gated by capital formation, construction, and a supply chain built at human pace. You cannot think a fab into existence. Worse, validating a new design means taping it out and testing it in silicon — a fabrication cycle measured in months — because a chip's real behavior at the edge of physics is not something you can fully simulate; that is the whole reason tape-outs exist. The recursion's clock is the fab's clock.

The AI that does science. Here the loop is intelligence improving the world model that makes it more capable — and much of frontier AI's near-term value is scientific, from materials to biology. The pitch, in the spirit of LeCun-style JEPA world models and model-based RL, is that a system with a good enough internal model of reality can plan and discover largely in simulation, then only occasionally check against the world. The catch is that internal state and reality drift apart, and the only thing that re-anchors them is an experiment. In biology the experiments are slow and, worse, noisy: a cell assay takes days to weeks, an animal study months, and the labels that come back are confounded and often non-reproducible. I have argued the specific mechanics of this elsewhere — the bottleneck in AI drug discovery isn't the model, it's the ground truth — and it generalizes. A superhuman scientist that never touches reality is a superhuman generator of hypotheses, which is not the same as a generator of knowledge. Autonomous "self-driving labs" are the honest attempt to industrialize this, and they help, but they run into the sim-to-real gap and into plain physical throughput: a robot still has to pipette, incubate, and wait, and incubation does not speed up because the scheduler got smarter. The recursion's clock is the assay's clock.

The AI that trains its successor. The most immediate loop: a model that improves the training of the next model, through better data curation, architecture search, and synthetic data. This is real and already happening at the margins. It is also the one most nakedly gated by compute and data. A better training recipe still has to be run, and the run consumes GPUs that are fab-limited and energy that is grid-limited — frontier training clusters are increasingly bottlenecked on power interconnects and substation build-out, a constraint measured in years and megawatts, not thoughts. And the highest-quality human data is a finite stock being consumed faster than it is produced; synthetic data helps but risks distributional collapse when the model learns from its own tail. The recursion's clock is the compute build-out's clock.

Notice the pattern. In all three, the intelligence step is exactly the part that gets faster, and in all three it is not the part that sets the pace. The dream imagines a self-hosting compiler — the classic bootstrap, where a compiler compiles its own improved source and each pass is nearly instant because it all happens in the same substrate, symbols acting on symbols. Recursive self-improvement through the physical world is not that. It is a compiler whose every pass has to stop and wait for a new machine to be manufactured before it can run. The bootstrap works in software precisely because the loop never leaves the substrate. Intelligence improving the physical conditions of its own production has to leave the substrate on every pass, and that exit is the rate limit.

The strongest counterargument, taken seriously

The best objection is not that the bottlenecks are fake. It is that a sufficiently intelligent system attacks the bottlenecks themselves. It finds a chip design that wrings more compute out of existing fabs. It discovers a training method an order of magnitude more sample-efficient, sidestepping the data wall. It designs experiments so informative that one settles what would have taken a hundred, or a simulator good enough to shrink the sim-to-real gap. In Bostrom's terms, it doesn't just push harder against recalcitrance; it lowers recalcitrance. This is not a hypothetical — it is the actual history of science. Better instruments and better theory have repeatedly collapsed constraints that looked physical. Sometimes the smart system will do exactly this, and any honest version of the skeptical case has to grant it.

Grant it fully, and the bound survives, because relaxing a physical constraint is itself a physical act subject to the same world. The sample-efficient training method still has to be validated by training a model and measuring it. The better fab process still has to be developed and proven in silicon, which is a fabrication cycle. The more informative experiment still has to be run against reality to know that it is in fact more informative rather than merely more clever — and if the system could know that in advance without running it, it would already possess the very ground truth the experiment exists to obtain. Every move that lowers a barrier bottoms out in a physical action gated by the same throughput it is trying to escape. The ceiling moves. It does not vanish. And it moves at the pace of the world, which is the thing we were trying to get past in the first place.

There is a deeper reason to distrust the unbounded version, and it connects to how these forecasts are built. The intelligence explosion is almost always argued by extrapolation: capability has risen fast, so it will keep rising fast, so it will rise fast enough to outrun every constraint. An extrapolated trend with no mechanism is a description, not a law, and it tells you nothing about its own boundary conditions — which is exactly the argument I have made about why scaling is not a theory of intelligence. You cannot read "the constraints will be overcome" off a capability curve, because the curve contains no account of what the constraints are. The fast-takeoff forecast borrows the confidence of an empirical trend to make a claim the trend cannot underwrite: that the physical inputs will yield on the same schedule as the benchmarks. Those are different variables with different clocks, and conflating them is the original sin of the whole genre.

Watch the bottleneck, not the benchmark

Here is the reframe, and the concrete thing to watch for. If you want to forecast recursive AI progress, stop staring at model capability and start tracking the external inputs, because those set the rate. Four gauges, in rough order of how binding they currently are:

Input What to watch Why it gates the loop
Compute Leading-edge fab capacity, tape-out cycle times, EUV supply New designs and runs realize at fab pace, not thought pace
Energy Grid interconnect queues, power availability per site, MW built Training and inference are increasingly power-limited
Data High-quality data supply, synthetic-data reliability Recipes need data to run on; the human stock is finite
Experiment throughput Wet-lab cycle time, autonomous-lab throughput, sim-to-real gap Knowledge that touches reality re-anchors at the assay's clock

The forecasting move is simple: when someone shows you a recursive-takeoff scenario, ask which of these four they are quietly assuming scales with intelligence. The ones that don't are where the timeline actually lives, and they are almost always the ones the scenario declines to mention. A superintelligent chip designer waits on fabs. A superintelligent scientist waits on experiments. A superintelligent trainer waits on compute and power. In each case the ceiling is set by a resource built and moved at human industrial pace — capital-intensive and slow — and a mind of any speed still has to stand in that line.

None of this says superintelligence won't arrive or won't be transformative when it does. It says the shape of the arrival is set by the world's throughput, not the model's cleverness — a climb steepened by intelligence but rate-limited by fabs, grids, and reality's refusal to be simulated for free. Intelligence is the accelerable fraction. Everything that matters is in the fraction that isn't. You cannot think your way past a wet lab, and the recursion knows it even if its prophets don't.

Frequently asked questions

Isn't this just saying superintelligence is impossible?
No. The claim is about rate, not ceiling. A system can be vastly superhuman and still improve at a pace set by fab construction, energy build-out, and experiment cycle time rather than by its own clock speed. Fast unbounded takeoff and slow, externally-rate-limited progress can both end at superintelligence; they differ in slope, and the slope is the entire strategic question.
A smart enough system could invent faster experiments, better chips, cheaper energy — doesn't that dissolve the bottleneck?
Sometimes it moves the bottleneck; it does not remove the category. Relaxing a physical constraint is itself a physical act gated by the same world: a better fab process still has to be built and validated in silicon, a better assay still has to be run against reality to know it works. Cleverness can lower recalcitrance, in Bostrom's term, but the act of lowering it is not free of the constraint it targets. The ceiling moves. It does not vanish.
How would I actually use this to forecast?
Track the external inputs, not the benchmark. Leading-edge fab capacity and lead times, grid interconnect queues and power availability, high-quality data supply, and above all experiment throughput in the domains where progress requires touching reality. When you see a recursive-progress forecast, ask which of these it assumes scales with model capability. The ones that don't are where the real timeline lives.

Filed under Cross-Disciplinary Deep Essays. Where biology, computation, markets, and philosophy collide.

Essays like this, in your inbox.

Thoughtful essays. No spam. Unsubscribe anytime.

Cross-Disciplinary Deep Essays

"AGI by Year X" Is Unanswerable Until You Name the Definition

"Are we close to AGI?" is incoherent because AGI names at least four incompatible criteria that come apart in practice. Separate them and the timeline debate dissolves into concrete, checkable questions.

9 min read