The transformer's dominance rests on a single design choice, and that choice has a price tag written into its arithmetic. Full self-attention compares every token to every other token, so its compute and memory grow with the square of the sequence length. Everything interesting at the architectural frontier is an attempt to keep what attention buys you while escaping that quadratic tax — and to treat the current design as the endpoint is to mistake a local optimum for a law.
Let me be precise about the cost first, because the precision is the argument. Then I want to walk the real alternatives — state-space models, efficient-attention approximations, retrieval and external memory, and the hybrids that are quietly winning — without the hype that usually attaches to each. The claim underneath all of it: architecture is not settled. Betting the field is "done except for scale" is a bet that one specific mechanism keeps winning, and that bet deserves to be named as a bet.
The quadratic tax, shown
Self-attention lets every position build its output as a content-weighted sum over every other position. For a sequence of length n, that means computing an n-by-n matrix of pairwise scores. Double the context and you quadruple the attention compute. Go from 4,000 tokens to 128,000 — a 32x increase in length — and the attention term grows by 32², which is 1,024x. The jump from a 4K context to a hypothetical 1M-token context is a factor of 250 in length and roughly 62,500x in pairwise work. That is the wall bounding context length, inference cost, and the ability to handle streaming inputs that never end.
Autoregressive inference makes the streaming problem concrete. To generate each new token, the model attends to every previous token, so it caches their keys and values. That KV cache grows linearly in memory with the sequence, and the cost of emitting one more token grows linearly with how much came before — which integrates to quadratic total cost over a long generation. A model reading an infinite stream would need infinite memory to keep attending to all of it. Attention has no built-in notion of forgetting.
Two honest caveats, because the field has partly fought back. FlashAttention (Dao and collaborators, 2022) is an IO-aware implementation that never writes the full attention matrix to slow memory; it recomputes on the fly and brings the memory footprint down to roughly linear, with large practical speedups. But the number of comparisons is still order n², so compute stays quadratic — engineering moved the constant, not the asymptote. And modern long-context transformers lean on sparse and windowed attention that only attends locally or to selected tokens. These help, but they are already admissions that full attention does not scale — the same admission the more radical architectures are built on.
Why attention is worth taxing in the first place
Before surveying escape routes, respect what you are trying to replace. Attention's superpower is content-based random access. Any token can, in a single layer, reach back and pull information from any earlier token selected by what that token contains, not by where it sits. This is not a metaphor; it is what the mechanism computes, and it is why transformers do in-context learning at all. Anthropic's interpretability work on induction heads (Olsson and colleagues, 2022) traced a concrete circuit: a head that finds an earlier occurrence of the current token and copies what followed it, implementing "if the pattern was AB before and I just saw A, predict B." That associative, content-addressed lookup over arbitrary distances is exactly the operation fixed-size recurrent states struggle to reproduce. Any replacement has to earn back this capability, and most of the tradeoffs below are about how much of it you are willing to give up for efficiency.
State-space models: a recurrent state that scales
The most substantive alternative line is structured state-space models. The lineage runs through S4 (Gu, Goel, and Ré, 2022) to Mamba (Gu and Dao, 2023) and Mamba-2 (2024). The core idea borrows from classical control theory: instead of comparing all pairs, maintain a hidden state that you update as you scan the sequence one step at a time, the way a linear recurrent system evolves. Processing is near-linear in sequence length, and — this is the streaming payoff — the state is fixed size, so per-step memory is constant no matter how long the input runs. An SSM can, in principle, read forever.
The obvious objection to any recurrent model is that a fixed state cannot decide what to keep based on the content flowing through it, which is why earlier linear-time models lagged on tasks like selective copying. Mamba's central contribution was to make the state-space parameters input-dependent — the "selective" in selective state space — so the model can choose to remember or forget based on what it is currently seeing, while a hardware-aware parallel scan keeps it fast on GPUs. Mamba-2 then formalized a structured duality between these selective SSMs and a form of attention, which is theoretically satisfying: the two families are not alien to each other but points on a shared spectrum.
The tradeoff is real and worth stating plainly. Compressing all history into a fixed-size state is lossy by construction, so on tasks demanding exact recall of a specific earlier token — precise associative recall, verbatim copying over long spans — pure SSMs have shown genuine weaknesses relative to attention, and this remains an active empirical question rather than a settled defeat. The mechanism that gives them constant memory is the same mechanism that makes perfect long-range retrieval hard. You do not get random access for free.
Efficient attention: approximate the matrix
A second line keeps attention's form but attacks the n² directly by approximating it. Linear-attention methods (Katharopoulos and colleagues, 2020, who framed transformers as RNNs) replace the softmax with a kernel feature map, letting you reorder the computation so cost grows linearly rather than quadratically. Performer (Choromanski and colleagues, 2021) approximates softmax attention with random features; Linformer (Wang and colleagues, 2020) projects the sequence to a low-rank summary; RWKV recasts attention-like mixing as a recurrence you can run in either parallel or sequential mode.
These buy efficiency, and the cost is usually fidelity: an approximation of full attention is, by definition, not full attention, and the gap tends to show up precisely on the sharp, selective retrieval that softmax attention does well. The category matters for builders because it is the least disruptive option — same mental model, cheaper bill — but it inherits attention's philosophy of compressing nothing only in approximation, so it is rarely the strongest choice for genuinely long-range work.
Retrieval and external memory: move context out of the forward pass
The third direction reframes the problem entirely. If attending to everything is expensive, do not put everything in the context. Retrieval-augmented approaches keep a large corpus in an external index and pull in only the relevant chunks at inference time. RAG (Lewis and colleagues, 2020) retrieves passages and conditions generation on them; DeepMind's RETRO (Borgeaud and colleagues, 2022) interleaves retrieval from a database of trillions of tokens directly into the model's layers, letting a smaller model match much larger ones by offloading knowledge to an external store rather than to parameters.
The conceptual move is that "context" need not live inside the quadratic forward pass at all. It can live in a database that scales independently, queried by cheap approximate nearest-neighbor search. The tradeoff is that retrieval quality becomes a separate system to get right — what you fetch, how you chunk it, how the model integrates it — and the model's reasoning is only as good as the retriever's recall. But it decouples the amount of knowledge available from the compute per token, which is a different and often better scaling story than "make the window bigger."
Hybrids are quietly winning
The most telling development is that the frontier is not choosing. The strongest recent long-context systems interleave attention and recurrence. DeepMind's Griffin (De and colleagues, 2024) mixes gated linear recurrences with periodic local attention and matched strong transformers at scale with better efficiency. AI21's Jamba (2024) interleaves Mamba layers with attention layers and mixture-of-experts, shipping very long usable contexts at production scale. The design logic is exactly the tradeoff analysis above: let cheap recurrent or SSM layers carry the bulk of the sequence mixing, and spend a few expensive attention layers where precise, content-based lookup actually pays. You use random access where it earns its cost and linear scanning everywhere else.
| Direction | Scaling in n | Keeps content-based random access? | Best fit |
|---|---|---|---|
| Full self-attention | Quadratic | Yes, fully | Reasoning needing exact recall; moderate context |
| State-space (S4/Mamba) | Near-linear, constant state | Partially, via selectivity | Very long or streaming inputs |
| Efficient/linear attention | Linear-ish | Approximately | Cheaper drop-in for long context |
| Retrieval / external memory | Decoupled from context | Via the retriever, not the pass | Large knowledge, small window |
| Hybrid (Griffin, Jamba) | Mostly linear | Yes, in select layers | Long context plus precise recall |
The deeper point: a trend is not a law
Here is why this survey is more than a menu. The transformer is a local optimum — a design that happened to sit at a good point in the space of attention, hardware, and optimization circa 2017, and that a decade of tooling then entrenched. It is not a theorem that this is the best way to model sequences. When people say the field is "solved except for scale," they are quietly assuming this particular mechanism will keep being the thing that scales best, and that assumption is doing enormous unacknowledged work.
I have argued elsewhere that we should not confuse the scaling curve with an explanation of why it holds — that it is Kepler without Newton, a description of the past mistaken for a law with boundary conditions. The architectural version of that error is subtler and just as expensive. Even if you fully believe the scaling curve, which architecture rides it most efficiently is an empirical, moving question, and the answer has already shifted once toward hybrids for long context. Sutton's Bitter Lesson says general methods that leverage computation win; it does not say self-attention is the final general method. It is entirely consistent with the Bitter Lesson that the next design to leverage computation better is not a transformer at all.
The alternatives above are still conservative — they keep the goal of predicting the next token in a sequence. The genuinely radical departure is to change the objective, which is what world-model research aims at: learning a predictive model of an environment in a latent space rather than modeling text token by token, the direction behind LeCun-style joint-embedding predictive architectures. That is a bigger bet than swapping attention for a scan, and I flag it as forecast rather than fact — but it is the reason "beyond transformers" might eventually mean beyond next-token prediction, not just beyond n².
What to actually do with this
For builders, the operative move is unglamorous: match the architecture to the workload instead of defaulting to "transformer" as a synonym for "model." If your bottleneck is very long or streaming inputs — logs, audio, genomic sequence, documents you read once and summarize — benchmark a state-space or hybrid model, or a retrieval design that keeps most tokens out of the forward pass, against a long-context transformer on your data. If your bottleneck is reasoning that has to jump to arbitrary earlier tokens by content, attention remains the safe default and you should pay its tax knowingly. The failure mode is not picking wrong; it is never running the comparison because you assumed the question was closed.
The transformer won a round. It did not end the game, and the scoreboard already shows the next players warming up.