Large video-generation models are quietly turning into world models, and the transition is being described loosely enough that its central flaw gets waved through. To predict the next frame of a video well enough to fool a human, a model has to internalize that objects persist when they leave the frame, that unsupported things fall, that a rolling ball keeps rolling, that two solids do not occupy the same space. Scaled video prediction yields a rough model of physics, objects, and cause-and-effect as a byproduct — and the frontier is now making these models action-conditioned, which turns a passive predictor into an interactive simulator an agent can roll out and plan inside. That is a genuinely promising route to grounding. It is also where the honest trouble starts, because a model trained to make footage that looks physical is not the same as a model of what is physical, and that gap is the entire story.
Let me be precise about the mechanism first, because the promise is real and worth stating at full strength before I take it apart.
Why next-frame prediction builds a rough physics engine
The training objective is almost embarrassingly simple: given some frames, and sometimes text, predict the frames that follow. Minimizing that prediction error at scale forces the model to discover the regularities that actually govern how pixels change over time. A model that has not learned object permanence will hallucinate objects into and out of existence across an occlusion and pay for it in loss. A model that has not learned rough Newtonian dynamics will mispredict where a thrown object lands and pay for that too. So the compression pressure of predicting future frames, applied over enough data, pushes the network toward an internal representation that behaves like an approximate model of the world's dynamics. This is not mysticism; it is the same logic by which next-token prediction on text forced language models to absorb a great deal of structure about how the world is described. Predict the future well enough and you are forced to model the thing generating the future.
The lineage here is real and predates the current hype. Learning a model of an environment's dynamics and then planning inside it is the core of model-based reinforcement learning, going back to Sutton's Dyna around 1990 and running through DeepMind's MuZero, which learned a model predicting only the quantities needed for planning and used it to master Go, chess, shogi, and Atari without being told the rules, and through Hafner's Dreamer line, which trains behaviors by rolling out a compact latent world model "in imagination." The video-native versions push in the same direction from the perception end rather than the state end. Yann LeCun's JEPA program is an explicit argument that the right objective is prediction in a learned representation space, not raw pixels, precisely so the model spends capacity on the predictable structure of the world and not on unpredictable texture. I have argued the general case for this architectural bet elsewhere — that the next real capability jump may come from a world model rather than a bigger LLM — and video is the most data-rich on-ramp to it we have.
The step that converts a video model from a party trick into an agent's tool is action-conditioning. A vanilla text-to-video model predicts a plausible future, but you cannot steer it frame by frame. Make the prediction conditional on an action — a control input, a button press, a steering command — and you get a system where you supply an action and it returns the next observation. Now you can do the thing planning requires: roll it forward under different action sequences and compare the imagined outcomes. This is what people mean when they call these systems "neural game engines": a learned, interactive simulator with no hand-written physics, no explicit geometry, just a network that answers "if I do this, what do I see next." Research demonstrations of playable environments generated entirely by such models, responding to inputs in real time, already exist. An agent that can roll out cheap imagined futures inside a learned simulator, instead of paying for every trial in the real world, is exactly the object that makes sample-efficient planning tractable.
That is the promise, stated as strongly as I can state it honestly. Here is the problem.
The objective rewards looking real, not being real
The loss function of a video model measures how close its output is to real footage — pixel error, a learned perceptual distance, or an adversarial critic's ability to tell fake from real. Every one of those is a measure of visual plausibility. None of them is a measure of physical correctness. The model is optimized to produce frames a discriminator or a metric judges as realistic. It is not optimized to conserve momentum, preserve object count, or respect causality — except insofar as violating those things happens to look wrong to the judge.
Most of the time that caveat carries the model, which is why the outputs are so impressive. Gross violations of physics usually do look wrong, so the training signal punishes them. But "usually looks wrong" is a leaky proxy, and it leaks in a specific, predictable pattern: the model can produce footage that passes visual inspection while violating the underlying invariants in ways a human eye scanning at frame rate will not catch but a planner relying on the rollout will. An object behind an occluder can reappear with a slightly different count or identity. A collision can resolve into a rebound that looks convincing but silently fails to conserve momentum. A stack of objects can settle in a way that reads as stable while the implied forces are nonsense. A long rollout can accumulate small dynamical errors that never look like a glitch but drift far from any physically realizable trajectory. The clip looks right. The physics is wrong. And the errors concentrate exactly where the visual prior is thinnest and the physical stakes are highest — occlusion, contact, force, mass, long horizons, rare events.
This is the same proxy-versus-truth failure that shows up everywhere in machine learning, wearing a new costume. The system optimizes the thing you can measure — does it look real — and you hope it delivers the thing you actually want — is it real — and the two agree until they don't. What makes the video case sharp is that visual realism is an unusually seductive proxy. A demo reel is viscerally convincing in a way a benchmark table is not. You watch it and your own visual system, which is very good and very fast, signs off. That felt conviction is precisely the thing to distrust, because your visual system is judging the same quantity the model optimized — plausibility — and neither of you checked the conservation laws.
A video world model is a surrogate, and its validity is the open question
This connects directly to a problem I have written about in the context of scientific computing: using simulation as a surrogate for experiment, and where that silently fails. A learned surrogate is trustworthy only in the regime where it was validated, and it fails most dangerously in the regime you most want to use it — the one outside the training distribution, where the interesting cases live. A video world model is a surrogate in exactly this sense. Its validity as a stand-in for real physics is not something the demo establishes; it is the entire question the demo dodges. When you plan inside a learned simulator, you are betting that its rollouts track reality closely enough that a plan good in the simulator is good in the world. That bet is a claim about validation, and "it looks photoreal" is not evidence for it.
There is a real counterargument, and it deserves its full weight rather than a strawman. It runs: visual plausibility and physical correctness are not independent axes — they are heavily correlated, because the fastest way for a scaled model to reliably produce plausible footage across an enormous distribution of scenes is to learn something close to the real dynamics. You cannot fake gravity across millions of varied clips without essentially discovering gravity. On this view the physics falls out of realism at scale, the residual errors shrink as models grow, and worrying about conservation laws is fighting the last war. This is the strongest case for optimism, and it is partly right: the correlation is real and it is why these models work at all. But correlated is not identical, and the argument proves less than it claims. Scale drives down average prediction error on the training distribution. It does not guarantee that the specific invariants a planner depends on hold in the specific rare or out-of-distribution states where planning decisions turn. A model can get monotonically more photorealistic while its momentum errors on edge-case collisions stay stubbornly load-bearing, because nothing in the objective ever specifically rewarded conserving momentum. Realism buys you correctness on average and abandons you at the tails, and the tails are where a surrogate earns or loses its keep.
The forecast, and the one question to ask
Here is my forecast, labeled as a bet rather than asserted as a result. Video world models will become genuinely powerful for training and planning in domains where visual plausibility correlates tightly with physical correctness — visually rich navigation, perception pretraining, tasks with short horizons and low-cost, correctable errors, environments that are forgiving about precise contact and force. In those regimes a cheap learned simulator beats expensive real experience and the residual physics errors wash out. And they will be actively dangerous in the domains where realism and correctness decouple — precise manipulation and contact dynamics, anything gated on mass or force, long-horizon rollouts where small errors compound, and rare safety-critical events, which is unfortunately much of what makes a simulator valuable for robotics and science in the first place. The uncomfortable pattern is that the decoupling is worst exactly where the payoff would be highest, the same shape as the surrogate problem in scientific computing. I hold this with real but bounded confidence: I could be wrong about how fast scale narrows the tail errors, and if the optimists are right it narrows faster than I expect. That rate is the number to watch.
Which gives one concrete, portable move. When someone demos a video world model — a neural game engine, a learned simulator, a "physically accurate" generator — ask whether it was validated on physical correctness or only on visual realism. Push for specifics: which quantities were measured against ground truth — positions, velocities, object counts, conserved quantities, contact forces — over what horizon, in which regimes, including the out-of-distribution and rare ones. A low perceptual-distance score and a gorgeous reel answer the realism question and are silent on correctness. If looking convincing is the only evidence on offer, what you are holding is a plausibility engine, and its fitness for planning is unestablished — not disproven, unestablished.
Predicting pixels that look physical gets you a rough world model for free. Getting one that is physical, in the states where being physical actually matters, is the work no amount of realism does for you.