Dreams That Last
A world model built on real SO-101 pick-and-place data. Give it five frames of context and then nothing but the joint commands, and it paints sixty steps of robot motion it has never seen. The lecture is really about one design decision: what the model carries between timesteps. We built the two obvious answers, watched both fail on the real robot, and then built the one that works.
The build, step by step
The exact path we followed — and the one you'll follow in the code kit.
The failure that started it
Our previous world model predicted the next frame almost perfectly — and then dissolved within a handful of steps once it had to feed on its own predictions. One-step accuracy and long-horizon stability are different problems.
What must a model carry?
Watch the gripper close on the cube: for about half a second the cube is hidden behind the hand. Its position is in no pixel and no joint angle. Something has to hold it — and "held" is not a number, it is a range. A world model tracks a belief, not a state.
Two designs, opposite failures
Design A carries one vector and never samples it: it remembers perfectly, but at a fork it must average both futures into one that never happens (error 0.409). Design B samples the state every step: it can fork honestly, but nothing survives the journey (2.598 — its error rose during training while its KL cost ballooned).
The merge
The RSSM stops choosing. It carries both: a deterministic memory that is never sampled, for facts, and a stochastic sample drawn every step, for doubt. The frame is painted from the pair — and because the sample feeds the next memory, a doubt once resolved becomes a remembered fact. Error 0.009, at the same parameter count.
We opened the memory up
Rather than assert that a recurrent state "carries information", we probed it: a plain linear readout recovers 99.9% of every joint angle from those 256 numbers, against a shuffled control at zero — and keeps recovering them forty steps after the camera is switched off.
And what we could not show
We expected uncertainty to spike at contact, where the future genuinely forks. Averaged over the state's dimensions, it does not. The t=0 collapse is real and measured; the contact spike is not there. We left that in the lecture rather than quietly dropping the plot.
From the run
