Every serious claim about an AI takeoff — fast or slow, dream or nightmare — traces back to a single paragraph written by I.J. Good in 1965. The paragraph is short, the reasoning inside it is a recursion, and almost everyone who invokes it has quietly upgraded a conditional into a prophecy. Read Good precisely, before decades of science-fiction sediment settled on him, and the argument says something narrower and more honest than the culture built on top of it: an intelligence explosion follows only if the returns to self-improvement do not diminish, and only if intelligence is the binding constraint on building the next machine. Both conditions are exactly what is in doubt. The explosion is not a theorem. It is a hypothesis with checkable premises, and the premises are where the whole question lives.
I want to reconstruct the argument as it was actually made, then take it apart at the joints, because the sloppiness here is expensive. Trillions in capital and a good deal of genuine fear are anchored to a claim most people have never read in the original and could not state as a conditional if asked. So let us read it.
What Good actually wrote
The source is a 1965 paper, "Speculations Concerning the First Ultraintelligent Machine," published in Advances in Computers. Irving John Good — Bletchley Park cryptanalyst, Bayesian statistician, later a consultant on 2001: A Space Odyssey — put the entire case in one passage:
Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an "intelligence explosion," and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.
Notice the shape. The argument has exactly one premise doing real work: the design of machines is itself an intellectual activity. From there it is pure iteration. If a machine surpasses humans at all intellectual activities, and designing machines is one of them, then it surpasses humans at designing machines, so it can design a successor better than itself — and that successor, being better still, designs a better one again. The recursion is the argument. Everything dramatic in the twentieth-century imagination follows from turning that crank.
Two things in the passage usually get lost. The first is the word "unquestionably," which is the load-bearing sleight of hand. Good asserts that the recursion explodes; he does not show it. The second is the closing clause — "provided that the machine is docile enough to tell us how to keep it under control." Good, in 1965, had already put his finger on the control problem Bostrom would formalize half a century later. He was not naive. He was compressed. And compression is where conditionals go to die.
The recursion is only as strong as its return function
Strip the drama and the argument is a claim about a sequence. Let the first ultraintelligent machine produce a design improvement — call the size of that first jump 1, in whatever units you like for "capability added." The machine's successor is better, so it produces some improvement too. The only question that matters is how big the second jump is relative to the first. Call that ratio r: the marginal return to being one generation smarter. What happens to the total depends entirely, and I mean entirely, on r.
There are three regimes, and they are not close.
| Regime | Condition | Improvement sequence | Result |
|---|---|---|---|
| Explosion | r > 1 | 1, r, r², r³, … | Diverges; super-exponential, a runaway |
| Steady growth | r = 1 | 1, 1, 1, 1, … | Unbounded but linear; a good tool, not a god |
| Fizzle | r < 1 | 1, r, r², … | Converges to a finite ceiling of 1/(1−r) |
The arithmetic of the third row is the one nobody quotes. Suppose each generation, being smarter, is nonetheless able to contribute only 90% as much new improvement as the generation before it — because the easy gains got taken first, because each further increment of intelligence is harder to wring out than the last. Then the improvements run 1, 0.9, 0.81, 0.729, and so on, a geometric series that sums to 1/(1 − 0.9) = 10. The recursion runs forever and buys you, in total, ten times the first jump. Then it asymptotes. That is a real and even impressive gain — a 10x — and it is still a fizzle. The system self-improves without limit in time and converges to a hard ceiling in capability. "The last invention man need ever make" becomes "a very good invention that plateaus."
Now hold r = 1. Each generation adds the same increment the last one did. Capability grows without bound, but linearly, at a walking pace: no explosion, no singularity, just steady compounding help. Only in the top row — where a smarter system produces a proportionally larger jump than it received, r above 1 — do you get Good's runaway. In continuous form that regime is even more violent than the table suggests: if the rate of improvement scales faster than linearly with current intelligence, the solution reaches infinity in finite time — a genuine mathematical singularity, which is where Vinge's word comes from.
Good simply assumed the top row. "Unquestionably" is the assumption that r > 1, asserted rather than argued. There is no law of nature that fixes r. Its value is an empirical fact about a specific kind of engineering problem, and it could sit in any of the three regimes.
Bostrom named the crux; he didn't resolve it
The most useful formalization came from Nick Bostrom in Superintelligence (2014), and it is worth stating because it is the same idea in cleaner clothes. Bostrom writes the rate of change of a system's intelligence as
rate of increase = optimization power ÷ recalcitrance
where optimization power is the effort applied to improving the system and recalcitrance is how hard the system is to improve. Run the recursion through this. As the system gets smarter, it can apply itself to the problem, so optimization power rises with intelligence. If recalcitrance stays constant or falls while optimization power climbs, the ratio grows, and you are in the explosion regime — Bostrom's fast, or hard, takeoff. If recalcitrance rises just as fast as optimization power, because each further gain in intelligence is proportionally harder to achieve, the ratio stays flat or shrinks, and you get slow takeoff or a fizzle.
This is my r by another name. "Recalcitrance rising as fast as optimization power" is exactly "r < 1." Bostrom's genuine contribution was to locate the crux precisely: the entire fast-versus-slow-takeoff debate is a debate about the shape of recalcitrance as intelligence increases, and nobody has measured that curve because the systems that would let you measure it don't exist yet. Vernor Vinge, in his 1993 essay "The Coming Technological Singularity," gave the phenomenon its modern name and its horizon-of-unpredictability framing. Ray Kurzweil, in The Singularity Is Near (2005), wrapped it in a "law of accelerating returns" and extrapolated a specific date. Naming a curve and dating its knee are not the same as showing the curve bends upward. The word "singularity" itself predates all of them — Stanisław Ulam, recalling a 1950s conversation with John von Neumann, reports von Neumann speaking of an "essential singularity" in the progress of technology. It has always been a suggestive image looking for a mechanism.
Why r might not exceed 1: the returns could diminish
Here is the first place the favorable assumption comes under real pressure. There is a respectable prior that hard problems get harder as you climb, not easier — that intelligence is one of the domains where the low-hanging fruit is picked first.
Consider what "design a better mind" actually entails as you go up. The first order of magnitude of improvement might be sitting in obvious architectural inefficiencies. The next might require insights that are genuinely scarce, the kind that took the whole field decades to find the first time. If the difficulty of the n-th increment of intelligence grows faster than the intelligence available to attack it, r stays below 1 and the whole thing converges. We have a name for this pattern everywhere else in research: diminishing returns to effort in a maturing field. Drug discovery has Eroom's law — inflation-adjusted cost per approved drug rising for decades as the tractable targets are exhausted. Fundamental physics has spent decades and enormous budgets consolidating a model largely fixed in the 1970s. Nothing guarantees recursive mind-design escapes the gravity that pulls every other hard research program toward a plateau. It might. But "it might" is a bet on the value of r, not a derivation of it.
The counter-case is real and I will not caricature it: software has no manufacturing step, a mind that improves its own algorithms can in principle redeploy the improvement instantly, and there may be a large overhang of capability reachable by reorganization alone before any new physics is needed. That is a serious argument for r above 1 over some range. It is also, precisely, an argument about a range — a burst of fast returns until the reorganizable slack is spent, which is the early, steep part of a sigmoid, not a proof of unbounded ascent. You cannot tell an exponential from the first bend of an S-curve using data from inside the bend, which is the whole problem I argue at length in Scaling Is Not a Theory of Intelligence: an unexplained upward trend is not a law with known boundary conditions, and reading destiny off it is a category error.
Why intelligence might not be the binding constraint at all
The second condition is the one Good's framing hides most completely, and I think it is the more likely place the argument breaks. Even grant r > 1 in pure design cleverness. The recursion still only explodes if cleverness is the bottleneck. It is not obvious that it is.
Walk through what generation n has to do to produce generation n+1. It has to have a better idea — fine, stipulate it does. Then it has to instantiate that idea in an actual machine: fabricate chips, or acquire the compute, or run a training process. Then it has to verify that the successor is genuinely better and not subtly broken, which is a hard problem in its own right, because evaluating a mind more capable than yourself is not something intelligence alone obviously solves. And if the improvement depends on any fact about the world the system does not already possess — a materials property, a biological mechanism, how a market or a protein or a plasma actually behaves — it has to run an experiment and wait for physical reality to answer.
That last clause is the killer. Experiments run at the speed of the world, not the speed of thought. A cell culture takes the time a cell culture takes. A chip tapes out on the fab's calendar. A superintelligence that needs a wet-lab result to design its successor is throttled to the clock rate of the wet lab, and no amount of additional IQ speeds up the incubator. This is the empirical-bottleneck problem, and it is why the sim-to-real gap matters so much in robotics and autonomous labs: the map you can compute is not the territory you must measure. If the rate-limiting input to building a better mind is data from the physical world rather than raw reasoning, then intelligence is not the binding constraint, and making the system smarter does not lift the ceiling — it just makes it wait more impatiently.
Good's argument implicitly assumes a world in which thinking harder is sufficient. The real world charges for information by the experiment, and the bill is paid in wall-clock time. The point here is structural: "intelligence is the binding constraint" is a second, independent premise, and it is doing as much work as the return function. The bottleneck, the verification problem, the economics of the compute the recursion consumes, and the safety question Good flagged in his final clause — including whether a self-modifying system even preserves its own goals across rewrites — each deserves its own treatment. Here I only need the joint to be visible.
Treat it as a hypothesis, not a legend
The honest way to hold all of this is to demote the intelligence explosion from prophecy to hypothesis — a claim with specific, checkable conditions, in the sense Popper would insist on. State it as a conditional and it becomes something you can actually reason about and watch:
If the marginal returns to recursive self-improvement stay at or above unity over many generations, and if intelligence rather than physical experiment is the rate-limiting input, then an intelligence explosion follows. Otherwise you get steady growth or a plateau.
Written that way, the forecast stops being a matter of temperament — techno-optimist versus doomer — and becomes a matter of evidence about two curves: the shape of r, and the location of the real bottleneck. Those are things you can look for. Watch whether each new generation of AI capability requires disproportionately more input than the last to achieve a comparable jump; that is r drifting below 1 in plain sight. Watch whether frontier progress is gated by ideas or by the availability of compute, data, and physical experiment; that tells you which premise is binding. The debate has better instruments than it uses, because it keeps arguing about the conclusion instead of measuring the premises.
It also helps to remember that recursive self-improvement is not a novel or mysterious thing we are meeting for the first time. Compilers compile their own next versions; a language bootstraps itself once a minimal seed exists. Science is a method that has spent four centuries improving its own methods. Both are genuine self-improving recursions with real history, and I take up what their track record actually shows — steady compounding, not runaway — in Recursion Isn't New. The base rate for the self-referential improvement processes we have already observed sits closer to the middle row of my table than the top one. That is not proof the AI case will rhyme. It is a reason to make whoever claims the top row earn it.
I.J. Good gave us a sharp and important idea and one unearned adverb. Delete "unquestionably" and the argument is still standing, but it is now a question — the right question, precisely posed. Whether the first ultraintelligent machine is the last invention we need make depends on a number no one has measured and a bottleneck no one has located. Anyone who tells you the answer is settled is quoting the adverb, not the argument.