MIRA — Making Multiplayer World Models Stable by Removing Hidden Information | Paper of the Week

This week we discussed MIRA: Multiplayer Interactive World Models with Representation Autoencoders, a technical report from General Intuition, Kyutai and Epic Games that trains a neural network to be the game engine for 2v2 Rocket League. The model watches 10,000 hours of gameplay and learns to generate what each of the four players sees, conditioned on what all four players press on their keyboards. It runs in real time, at 20 frames per second on a single Nvidia B200 GPU, and its rollouts stay coherent for hours.

The report was two days old when we met: 60 pages, plus a blog version whose rollout videos show most of the behaviour we discuss below. We picked it partly because General Intuition's name had just come up in an investment conversation, and we wanted to understand what these generative world models actually do.

The paper's headline is an existence claim: the first real-time multiplayer world model for a genuinely dynamic physical environment. The finding we cared about most sits underneath it: a multiplayer world model, one that conditions on every player's actions and renders every player's view jointly, models a shared multi-agent world more faithfully than a single-player model of the same match, which has to treat the other agents as scenery. Why that works turned out to be the most useful intuition of the call.

What MIRA is, in one pass

MIRA has two parts. A codec compresses each player's video into a compact latent, roughly 192× smaller than the raw pixel stream, built on a frozen DINOv3 feature extractor. A latent world model, a 5-billion-parameter diffusion transformer, then predicts the next latent frame from the past latents and all four players' actions, ten latent frames per second, and the codec's decoder turns each prediction back into the 20 fps video the players see. The model gets nothing privileged from the game engine; it learns from pixels and key presses alone.

Two facts about the setup are worth knowing before you watch the demos. First, there is no human data at all: every car in every match is driven by Nexto, a publicly available Rocket League bot, playing against copies of itself. Second, the environment is nearly static: all matches happen on three fixed arenas, with no scenery variation beyond them. What the model has to learn is car and ball physics under player actions, not open-ended world generation. That is a far more bounded problem than the "generate any world" framing this research area attracts, and we think the boundedness is doing a lot of the work in the results.

Why four views beat one

The result we kept returning to is that a shared world model for all four players stays self-consistent where a single-view model diverges.

The paper shows this from both directions. Train on a single player's view and the rollouts degrade in characteristic ways: a car that leaves the frame is often gone when the camera returns, and occasionally a car sitting beside the ball is absorbed into it and never re-emerges. Train on all four views jointly, tiled into one grid so spatial attention spans every perspective, and those failures largely disappear. You can even watch the mechanism in the paper's cross-view attention maps: a query on a car in one view lights up the same car in the other three.

The mechanism is hidden causes. From one player's view, the other cars act off-camera, so the model has to render effects whose causes it never observes. It is forced to imagine three hidden agents, and it never fully disentangles "what I did" from "what the unseen players must have done"; that inference gap is exactly where its hallucinations live. With all four views and all four action streams there is almost nothing left to infer. Rocket League is nearly deterministic once every player's inputs are known, so predicting the next frame collapses into learning how actions map to consequences.

There is a good training-recipe detail here too. Multiplayer from scratch collapses at the paper's compute budget; what works is pretraining on single-player views and then continuing on the four-view layout. And during training, players' action streams are randomly withheld, so the model learns to drive uncontrolled cars itself. The paper calls this an emergent, limited theory of mind, which seems fair.

An analogy that fell apart: chess and poker

On the call we reached for a game analogy: the single-view model is playing poker, hidden hands everywhere, while the multiplayer model plays chess with the whole position on the table. Writing this up, we realised the analogy - while somewhat interesting - did not fully survive scrutiny.

The model is not a player. Players sit inside a game; this model's job is to actually be the game. It takes everyone's inputs and paints everyone's screens. It is the engine.

And so, the analogy with the games mentioned would have to be being able to accurately predict future states - i.e. the next player's move.

In chess, watching the board is watching everyone's actions: moves arrive one at a time, each fully legible from the state change and extremely basic. We essentially have complete information and nothing other than being in the players' heads can further help predict future states.

In Poker, however, we can view things in two distinct ways:

From any player's perspective: which maps to MIRA's single player model - predicting future states is very hard (and partly what makes the game of poker so interesting) as we don't know what cards the other players are holding. As such, the model would struggle immensely to predict any realistic future states let alone actual future states.
From the game engine's perspective, where it knows all the cards being help by all the players + the deck: predicting states with this knowledge is much easier. This perspective is the closest there is to the multiplayer's model view. It allows the game engine to predict realistic future states - that is, states where once the players cards would be shown, wouldn't surprise us. The engine could even be running a game theory optimal (GTO) strategy for each player in each state to try and minimise surprise.

Although the second poker perspective can be viewed as perfectly mapping to the multiplayer Rocket League model from MIRA, because we don't know what lies in the player's heads, our predictions will likely mostly fall flat. MIRA and Rocket League don't have that problem, they are just trying to predict frames. A bit like if the poker engine were trying to predict what the screen should look like once player A bet x$ - a rather trivial thing to do in poker.

So really, what we concluded was that the analogy was only somewhat useful when thinking about a complete information poker game trying to predict future states - but even then, we still have too many unknowns. On top of that, the screen painting problem is not an interesting one in the games mentioned.

Multi-agent conditioning pays off exactly when actions are simultaneous, partially observable, and physically entangled. Environments like that, where the complexity lives in the dynamics rather than in hidden intentions, are also where these models fit best.

A frozen encoder, then a generative decoder

Training happens in two separate passes, and the split matters. First the codec is trained on video alone, with no actions anywhere, purely to reconstruct frames; the paper's codec section details the losses, and the notable part is what's missing: no adversarial term, no injected noise, no KL, a deterministic latent. Then the codec is frozen. The world model trains entirely inside its latent space with a flow matching loss, and the decoder only reappears at inference to render frames.

We spent a while contrasting this with the JEPA line of work. V-JEPA also encodes video into a latent space and predicts there, but it deliberately never reconstructs pixels: the representation is the product, and planning is meant to happen in latent space. MIRA makes the opposite bet: because the product here is a playable video stream, it keeps a full generative decoder, a causal space-time transformer with a temporal upsampler that draws every pixel.

Two kinds of time

The most mathematical stretch of the call was about the generative objective, and most of the confusion dissolves once you keep two clocks separate.

One clock is frame time, $t$ : the ordinary index over latent frames. The other is flow time, $\tau$ , which runs from 0 to 1 for each frame individually and has nothing to do with the video's temporality. At $\tau = 0$ a frame is pure Gaussian noise $z_0$ ; at $\tau = 1$ it is the clean latent $z_1$ ; in between it sits on the straight-line interpolation

z_\tau = \tau\, z_1 + (1 - \tau)\, z_0.

The model is not trained to predict the clean latent directly. It predicts the velocity of the denoising path - which is the standard flow mathcing approach. The training loss, the equation we spent the longest on, is

\mathcal{L} = \mathbb{E}_{\tau,\, z_0,\, z_1}\, \bigl\lVert v_\theta(z_\tau, \tau) - (z_1 - z_0) \bigr\rVert_2^2,

an expectation over the flow time, the noise and the data. Parameterising the derivative of a continuous path is more tractable than learning one giant jump from noise to image, and because the target paths are straight, inference is just first-order integration: start from noise and repeatedly step along the predicted velocity, $z_{\tau+\Delta} = z_\tau + \Delta\, v_\theta(z_\tau, \tau)$ , until $\tau$ reaches 1 and a clean frame drops out.

The trick that makes rollouts stable is diffusion forcing: every frame in a training clip draws its own independent $\tau$ , so 20 latent frames means 20 denoising paths at 20 different noise levels, some nearly clean, some nearly destroyed. That mixture is exactly what the model faces at rollout, where it conditions on its own imperfect predictions. The ablation is stark: teacher forcing collapses roughly tenfold once the rollout passes the 4-second training window; diffusion forcing stays essentially flat out to five minutes, the longest horizon measured.

What real time costs

Getting this to 20 fps takes real engineering: the model is distilled so one or two network evaluations replace the whole flow-matching integration, inference keeps a rolling window of just 20 latents with a streaming KV cache, and a full step (world-model update plus decoding) lands around 70 ms for two frames. The paper's streaming-inference section covers the rest of the systems work.

Even so, that is one player-facing stream at 20 fps saturating a top-end datacenter GPU. We compared it to Genie on the call: the headline frame rates in this space show up in far more basic worlds, and MIRA's rate is less surprising once you notice it only has to learn the behaviour of cars in a fixed arena. The conclusion we settled on is that diffusion world models currently fit environments of bounded complexity: a fixed arena, one game's physics. The compute and stability story for large, variable worlds is not there yet.

Where it breaks: a context shorter than a match

Twenty latents is two seconds of memory, and most failure modes trace back to that or to imbalance in the training data.

Goal replays are the clean example. After a goal the game cuts to a scripted replay, but a full replay outlasts the context window, so the model invents one: the replay never shows the goal that was actually scored. The clock and score drift the same way. Data imbalance shows up as physics priors: a ball left untouched tends to gain speed and roll toward a goal, because a resting ball almost never occurs in recorded play, and the model boosts off the kickoff even when the player presses nothing, because the bots always did. The paper's failure-cases section catalogues these honestly.

The out-of-distribution behaviour is stranger. Park your car in your own goal, something the competitive bots never do, and that view can desynchronise and degrade into noise. Then, untrained and unprompted, the model pulls it back: the views resynchronise on a scripted goal replay and coherent play resumes. The blog version shows more of this drift-and-reset behaviour, including cars blowing up with nothing touching them. To us it reads as further evidence that the joint four-view representation is the stabilising anchor.

From game engine to road simulator

We care about this line of work because it is a recipe for learning a simulator from data instead of hand-coding one, and simulators are how you train and evaluate embodied agents before letting them touch reality. Record enough interaction and you can, in principle, distill the environment itself.

Our pushback was about coverage. MIRA's whole failure catalogue comes from behaviours the data never contained, and for driving, the behaviours that matter most are exactly the unpredicted ones: the pedestrian who crosses without looking, the animal that runs into the street. A learned simulator built only from competent, competitive play would inherit MIRA's blind spots in the worst possible place. One idea from the call was modularity: a family of focused simulators, one for pedestrians, one for intersections, one for motorway driving, rather than a single model of everything, keeping each inside the bounded regime where these models demonstrably work.

And we would not put any of this near a physical robot yet. Generated video still lacks physical anchors: today's video models produce plausible frames with impossible physics, and a policy trained inside a simulator that fails on a few percent of unseen situations fails exactly where it is most dangerous. The paper is honest that visually plausible futures are not the same as correct responses to interventions; its evaluations (action adherence, physical-state probes) exist precisely because pretty frames hide the difference.

What we refined

A few things the call sharpened that the paper states only briefly:

The velocity target $z_1 - z_0$ is constant along each straight path, which is what makes few-step integration behave; predicting the endpoint directly makes the high-noise regime needlessly hard.
The notation fights the diffusion literature: there, $x_0$ is the clean image; here, $z_1$ is. We'd rename the endpoints (data and noise, say) before teaching this to anyone. The 80-page MIT lecture notes on flow matching we're circulating are a better on-ramp than reverse-engineering the convention from the loss.

The takeaway

MIRA is compelling because it makes a simple bet:

Show the world model every player's view and every player's actions, and there is no hidden information left. Predicting the future stops being a guess about unseen agents and becomes interpolation of the dynamics.

Once that is true, a diffusion transformer over a good frozen latent space really can behave like a game engine: coherent for hours, controllable by four people at once, in real time. The main intuition we took away is that the multiplayer conditioning made the problem easier, not harder: the extra views removed uncertainty instead of adding it. The equally important reading is about scope. This works because the world is small, and the failure modes (two seconds of memory, priors soaked in one bot's habits) are a precise list of what still separates a playable model of a game from a simulator you could train a robot in.