Whatever you believe about the intelligence-explosion timeline, one thing holds independent of the forecast: recursive self-improvement makes the alignment problem structurally harder, and it does so for two specific reasons that require no science fiction to state. First, nobody knows how to guarantee that a system rewriting itself preserves its own goals across the rewrite. Second, our ability to check that a system is aligned rests on being able to understand its behavior, and that ability falls behind exactly as each generation outruns the last. Both are recognized open problems, not speculative alarms, and both get worse in precisely the regime where being wrong costs the most.
I want to separate two questions the singularity discourse keeps fusing. The first is when, or whether, an AI system recursively self-improves into something vastly beyond us. The second is what happens to our ability to steer it the moment self-improvement starts at all, even slowly, even at human level. The first is a forecast, and a contested one. The second is a claim about mechanism that holds across almost any timeline. This essay is entirely about the second. You can be a hard skeptic about fast takeoff and still owe yourself an answer to it.
The idea is old, and its core worry was never the speed
I.J. Good stated the seed in 1965, in "Speculations Concerning the First Ultraintelligent Machine": an ultraintelligent machine could design still better machines, so "there would then unquestionably be an 'intelligence explosion,'" and the first such machine is "the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control." Read that last clause again. Good's own framing put the control problem inside the definition of the outcome. Vernor Vinge in 1993 and Ray Kurzweil in 2005 popularized the "singularity" and argued the timelines. Nick Bostrom, in Superintelligence (2014), did the careful work of separating takeoff speed — slow, moderate, fast — from takeoff dynamics, and it is Bostrom's framing that matters here: the danger is not principally that the thing is fast. The danger is that self-improvement is a process whose fixed point we cannot currently constrain.
Strip away the drama and recursive self-improvement is just a system in a loop, using its current capabilities to produce a more capable successor, which repeats. We already have a benign, fully understood instance: the self-hosting compiler. GCC is written in C and compiled by GCC; each version bootstraps the next — a real, working example of a tool improving the tool that builds it. It is also the source of the two problems, because a self-hosting compiler is exactly the setting of Ken Thompson's 1984 "Reflections on Trusting Trust," where a compiler is taught to insert a backdoor into programs it compiles and into future copies of itself, so the malicious behavior propagates forever and appears in no source code you can read. Thompson's point was epistemological: when a system participates in producing its own successors, source-level inspection stops being sufficient to know what the successor will do. Both of my two problems are that observation, made precise.
Problem one: a system that rewrites itself has no guaranteed grip on its own goals
State the problem exactly. An agent that can modify its own weights, code, or objective representation produces a successor, and there is no known method that guarantees the successor holds the same goals as the original. This is the goal-stability-under-self-modification problem, and it is genuinely open in the AI-safety literature, not a settled engineering detail.
The intuitive reply is that a rational agent would never want to change its own goals. Bostrom calls this goal-content integrity and lists it among the convergent instrumental drives, alongside Steve Omohundro's earlier "basic AI drives": an agent that values X has an instrumental reason to ensure its future self still values X, because a successor that valued something else would not pursue X. That reasoning is sound as far as it goes, and it is why the naive fear — the AI spontaneously deciding to want different things — is too crude. The agent is trying to preserve its goals.
The problem is that wanting to preserve your goals and being able to verify that a successor does are different things, and the gap between them is where alignment leaks. Preserving a goal across a self-rewrite requires the modifying system to establish that the modified system still optimizes the same objective. For any successor even slightly more capable or differently structured than itself, that guarantee runs into a wall that is formal, not merely practical. The research community around MIRI spent years on this under the headings of tiling agents and Vingean reflection, and hit what they named the Löbian obstacle: a sufficiently expressive formal agent cannot, in general, fully trust the conclusions of a successor that reasons with the same or greater power, because doing so requires it to assert the soundness of a system as strong as itself, which Gödel's and Löb's theorems block. You do not need to accept that formalism as the final word to take the lesson. "Just have the system verify that its rewrite is goal-preserving" is not a checkbox; it collides with the limits of formal self-reference.
Now add the part that makes it bite regardless of the theory: drift compounds. Suppose each self-modification preserves the original objective with probability p — the successor's values match to within tolerance. Across n generations, the probability that alignment survives intact is roughly p^n, because each step is another independent chance to slip. Put numbers on it. If every rewrite is 99.9% goal-preserving, a rate no current technique can promise, then after 1,000 rewrites the chance the system still holds its original goals is 0.999^1000 ≈ 0.37. At 99% per step, a hundred rewrites suffices: 0.99^100 ≈ 0.37. A self-improving system might run thousands of modification cycles in the time a human team reviews one. The per-step reliability required to keep long chains aligned is far beyond anything we can currently establish, and the number of steps is exactly the quantity self-improvement is designed to maximize.
This is a different failure from a model being biased or hallucinating. It is the failure of transitivity of trust: even if you fully align generation zero, you have not thereby aligned generation one thousand, and the mechanism that would carry alignment across the chain — a rewrite verified to be goal-preserving — is the thing we do not know how to build.
Problem two: verification degrades exactly as the system passes its verifiers
The second problem is the one I find more certain and more neglected, because it does not depend on any formal-logic obstacle at all. It is arithmetic about who can check whom.
Our entire practice of judging whether a system is aligned rests on a capability match: a human, or a tool we trust, has to understand the system's behavior well enough to evaluate it. We read the outputs, probe the edge cases, run the evals, inspect the reasoning. Every one of those methods assumes the checker can comprehend what the checked system is doing. Recursive self-improvement attacks that assumption directly, because the entire point of the loop is to produce systems that exceed their predecessors. Each generation is, by construction, harder to evaluate than the last — and it becomes harder to evaluate at the same time as it becomes more consequential to get right. The verifier-capability gap opens exactly where the stakes rise.
"Keep a human in the loop" is the reflex answer, and it degrades most precisely when you need it. A human reviewing a system modestly smarter than the last one can still spot-check. A human reviewing a system that reasons in ways they cannot follow is not verifying; they are rubber-stamping outputs they lack the capacity to contest, which is worse than no oversight because it manufactures false assurance. This connects to a constraint I have argued lives at the center of the whole agent question. The human in the loop is not there to compensate for capability — they are there to be the accountable party who answers for the outcome with real skin in the game. Self-improvement corrodes both functions of that human at once: it removes their ability to evaluate the system while leaving them nominally liable for it. You end up with a person legally on the hook for a process they can no longer understand, which is the exact shape of the trust crisis that opens as agents outrun our ability to verify who they are and what they optimize.
The tempting escape is to have AI verify AI: if humans cannot check the superhuman system, use another capable system to check it. This is where the self-verification regress bites, and it is the same structure I traced when arguing that generation is going to zero while verification stays expensive. Using a model to verify a model does not close the gap; it relocates it. Now you must trust the verifier, so you need a verifier for the verifier, and the regress either bottoms out in something a human can actually check — in which case you are bounded by human verification capacity after all — or it bottoms out in a system you trust because it is capable, which assumes the conclusion. Capability is not evidence of alignment. A more capable checker is also a more capable potential deceiver. The regress does not resolve by adding intelligence to it; intelligence is the thing in doubt.
None of this says verification is hopeless. It says verification has to be engineered to scale with capability, and that this is a hard, unsolved research program rather than a property we get for free. The serious responses in the literature are attempts to make oversight scale: Paul Christiano's iterated amplification, the debate proposal from Geoffrey Irving and colleagues, recursive reward modeling, and the interpretability work aimed at reading a model's internals rather than only grading its outputs. The honest status of all of them is: promising, partial, unproven at the capability levels that would matter most. That is the whole point. Whether verification keeps pace with capability is an open empirical question, and until it is answered you cannot assume the answer is yes.
The strongest version of the counterargument
Let me put the best case against my thesis, because it is a real one and a reader should hold it.
The counterargument runs like this. Self-improvement will not be a discrete event where a system rewrites its own weights in an unreadable leap. It will be gradual and tool-mediated. Systems will improve by writing better training code, curating better data, designing better successor architectures that humans still train and inspect — more like the messy, human-in-the-loop way software already improves than like a mind editing its own brain in the dark. Each increment will be small enough to evaluate with the tools of the previous generation, including AI tools that improve alongside the systems they check. On this view, capability and verification co-evolve, the verifier-capability gap never opens wide because the checkers ride the same curve as the checked, and goal drift is caught at each small step the way we catch regressions in a test suite. The likeliest world, the skeptic says, is not a runaway loop but a slow climb where oversight keeps up because it is improving too.
I take that seriously, and I think it is the most probable path. Notice what it concedes, though. It concedes that safety depends entirely on verification staying coupled to capability at every step — that the co-evolution actually holds, that the AI tools checking each generation are themselves aligned and do not share the blind spots of what they check, and that no one runs the loop faster than the checking can keep up. Those are not guarantees. They are the exact conditions that have to be maintained, and they are maintained by choice and engineering, not by nature. The counterargument is not a reason the problems are illusory. It is a description of the narrow regime in which they stay manageable. My thesis and the strongest objection agree on the operative claim: alignment under self-improvement survives only if verification is deliberately kept in pace with capability. The disagreement is merely about how likely we are to hold that line by default. Betting the high-stakes cases on "by default" is the error.
The principle: bind every autonomy grant to reversibility and to verification that scales
Here is what falls out, and it is a design rule you can apply now, before any of the speculative parts resolve. The autonomy you grant a system should be bounded by two quantities: how reversible its actions are, and how well your verification scales with its capability. Both, not either.
Reversibility is the cheap safeguard and the one self-improvement most threatens, because a system that can improve itself can take actions whose consequences you cannot undo, including changes to itself. So the rule for self-modifying systems specifically: do not deploy them where errors are expensive, irreversible, and beyond your capacity to verify, until verification demonstrably keeps pace with capability. That is not a moratorium on the research; it is a placement rule. Let self-improving systems run where the loop is fast, the actions are reversible, and the checks are machine-gradable — code that a test suite gates, simulation, sandboxed optimization. Keep them out of the irreversible, weakly verifiable, high-consequence positions where the two problems compound. This is the same logic that already governs model-based reinforcement learning and JEPA-style world-model systems in the physical world, where the sim-to-real gap is a standing reminder that a policy verified in simulation is not verified in reality, and where autonomous "self-driving labs" are deployed exactly where a wrong action wastes reagents rather than harms a person.
The concrete thing to watch for, in any system marketed as self-improving or self-optimizing: ask whether its verification story scales with its capability, or whether it quietly assumes a human or a fixed tool at the end of the chain who is falling behind. If the answer is a human in the loop, ask what that human can still actually check, and what happens to the guarantee on the day they can no longer follow the reasoning. If the answer is "another model checks it," ask who checks that one, and keep asking until you hit something you can ground or something you are trusting on capability alone.
The intelligence explosion may be near, far, or never. That question is genuinely open and I hold no strong view on the timeline. What is not open is the shape of the difficulty: self-improvement compounds goal drift across generations we cannot prove safe, and it outruns the verifiers we would use to catch the drift, and it does both worst where the cost of being wrong is highest. Those are reasons alignment is hard, not proof of doom — and the distance between the two is exactly the verification we have not yet built. Build it before you hand the system the ability to build itself.