The World Modeler's Toolkit
Four tools every world model stands on: compressing the world into a few numbers, reward over time, value, and the two ways to learn behaviour.
In the last lecture we built the picture that everything else in this series rests on: an agent and an environment, trading actions for observations, with the true state of the world hiding behind what the agent can actually see. Next lecture, we build our first learned world model — in code, from an empty file.
But between the picture and the code, four tools come first. Every paper in this field assumes you know them. Most never explain them. So today we build each one from zero, with no reinforcement learning background assumed.
None of this is busywork. Each tool answers a question that the agent-environment loop forces on us. You will see the question first, and the tool second. By the end, you will hold four ideas — compression, return, value, and actor-critic — and you will see why they click together like parts of one machine. This lecture is taught on the Vizuara YouTube channel as part of the Build a World Model from Scratch series; this article is the full written version.
Let us start with the tool that is about perception: turning a hundred thousand pixels into a handful of numbers that keep only what matters.
Tool 1: Compression — The Sketch-Artist Game
Here is a game you have seen in every police drama. A witness sits across a table from a sketch artist and describes a face — in maybe twenty words. "Round face, heavy eyebrows, a scar above the left eye, hair swept back." The artist draws. And the drawing comes out remarkably close to the real person.
Stop and think about what just happened. A face is a million pixels. It crossed the table as twenty words — and it survived. The description kept what mattered and dropped everything else: the exact shade of every skin pixel, the lighting, the background, all of it discarded, and yet the identity of the face made it across.
The two people in this game have technical names. The witness is an encoder: face in, compact description out. The artist is a decoder: compact description in, face out again. And notice the built-in quality check — the closeness of the redrawing tells you exactly how good the description was. If the sketch looks nothing like the suspect, the witness left out something essential.
Now replace the twenty words with numbers, and replace both people with neural networks.
The Autoencoder: Squeeze, Then Reconstruct
The neural-network version of the sketch-artist game is called an autoencoder, and it works in three steps.
Step one: the encoder squeezes. An image is forced through a narrowing network into a tiny vector — say 32 numbers. That vector is called the latent. It is the twenty-word description, written in the only language a network speaks.
Step two: the decoder redraws. From those 32 numbers alone — nothing else — a second network tries to redraw the original image.
Step three: the error is the teacher. Compare the redrawing to the original, pixel by pixel. Every mismatch flows backward through both networks as a lesson: keep more of what I lost. Notice that no labels are needed anywhere. The image grades itself.
The shape of the network is an hourglass — wide, then narrow, then wide — and the narrow middle is the whole trick. Make the bottleneck tight enough and the network has no choice but to discover what is essential about the image, because 32 numbers cannot carry anything else. Compression here is not a compromise we grudgingly accept. Compression is how the understanding happens. The bottleneck forces the network to separate what matters from what merely fills the frame.
Those 32 Numbers Form a Space
Here is where the latent stops being a storage trick and becomes something you can think in.
First, each of the 32 numbers tends to mean something. Train an autoencoder on faces, then take the latent of one face and turn a single number up: the smile widens. Turn another: the lighting shifts. The latent is a small control panel for the image — 32 sliders, each wired to something real.
Second, nearby points in this 32-dimensional space are similar images. Two faces that look alike land close together; two that look nothing alike land far apart. This is a quiet but profound shift: similarity has become distance — something you can compute with, not just squint at.
And third — this is the payoff for our series — that geometry is what makes prediction possible. "Predict the next frame of the game" sounds like an impossible task if you think in pixels: repaint a hundred thousand values, every one of them exactly right. But in latent space, the next frame is a nearby point. Prediction becomes a small, smooth step in a small, smooth space.
Why Every World Model Lives in Latent Space
Remember the plan from Lecture 1: a world model must predict the future over and over, thousands of times, to dream a single episode of experience.
Now run the numbers. In pixels, this is hopeless. A 64×64 colour frame is 12,288 numbers. Predicting all of them, at every imagined step, for a thousand steps — with most of the effort wasted on textures and background details that no decision depends on. In latents, the same job is routine: encode the frame once, predict 32 numbers per step, and decode back to pixels only when you actually want to look at the dream. Hundreds of times cheaper, and the model spends its capacity on what actually changes the future.
This gives us the recipe of the entire series, in one sentence: compress, then predict. Every system we will build — from the first world model next lecture all the way to Dreamer 4 — is this one sentence, refined.
Tool 1 is in the box. Now for the question that perception alone cannot answer: once an agent can see, how does it know what good looks like?
Tool 2: Reward Over Time — The Best Move Often Looks Bad Right Now
Consider two situations.
Chess: you give up your queen. By every immediate measure this is a disaster — you just lost your strongest piece. And sometimes it is precisely the move that wins the game five moves later.
Life: you study tonight instead of going out. The evening is a pure loss. The payoff arrives months away, on an exam.
Both situations teach the same lesson: never judge an action by its instant reward. Judge it by the total reward it leads to — everything from now until the end. That total has a name: the return.
return = r₍now₎ + r₍next₎ + r₍after₎ + …
The return, not the immediate reward, is the number an agent actually tries to maximise. Behaviour is not about grabbing the coin in front of you; it is about choosing between entire streams of reward.
The Discount: Tomorrow's Coin Is Worth Slightly Less
There is one refinement, and it comes with a Greek letter. We shrink each future reward by a factor γ (gamma) per step — say 0.99:
return = r₀ + γ·r₁ + γ²·r₂ + γ³·r₃ + …
γ sits between 0 and 1, so the further away a reward is, the smaller its weight. Why do this? Two reasons, one honest and one practical.
The honest reason: the far future is genuinely uncertain. The game may end; the world may change. A promised coin a hundred steps from now should count for less than a coin in your hand — and with γ = 0.99, a reward 100 steps away counts about 37% of one today. The discount is not pessimism; it is an accurate accounting of uncertainty.
The practical reason: without a discount, an endless game gives an infinite sum, and you cannot maximise infinity. Discounting keeps the total finite.
And notice that γ is a single dial with a single meaning. Set it near 0 and the agent is greedy — it lives for today. Set it near 1 and the agent is patient — it plans far ahead. One number controls the agent's entire attitude toward the future.
The Hard Part: Who Deserves the Credit?
So the agent maximises the discounted return. Straightforward — until you try to learn from it.
You win a chess game at move 40. Which of your forty moves caused the win?
The reward arrives late and says almost nothing. One "+1, you won" at the very end, and silence before that. The signal does not point at the brilliant move 12 that set everything up. It does not point at the blunder at move 30 that almost threw it all away. It just says: the whole trajectory, taken together, worked out.
This is the credit-assignment problem: spreading one late reward correctly across the many decisions that led to it. It is the central difficulty of learning behaviour — and it is the reason the next tool exists. We solve it by learning a number that fills the silence: a running estimate of how well things are going at every step, not just the last one.
Tool 3: Value — A Price Tag on a Situation
Why does a house near the park cost more than the identical house near the factory? Not because of anything happening today. The price is high because of everything living there will bring, added up — the mornings in the park, the quiet, the resale. A price is the future, summarised into one number, available right now.
That is exactly what a value function is:
V(s) = expected return, starting from state s
A learned price tag for every situation. The same idea runs through chess: a grandmaster glances at a position and says "White is winning" — long before any piece is taken. That judgement is a value estimate. It compresses everything that is likely to follow from this position into a single verdict about the present.
And here is how value solves the credit-assignment problem. Reward speaks once, at the end of the game. Value speaks at every step: did this move raise my prospects or lower them? Compare the value of the position before your move to the value after it, and every single move gets its own verdict. One late reward becomes forty per-move signals. The silence is filled.
Two Flavours: The Value of a Place, and the Value of a Choice
Value comes in two flavours, and you will meet both letters everywhere in this field.
V(s) asks: how good is it to be here? One number for the situation itself. The house's market price. The grandmaster's "White is winning."
Q(s, a) asks: how good is it to do this, here? One number per action: standing at a fork, what does each road lead to? And notice what Q buys you — if you know Q for every available action, acting becomes trivial. Take the biggest number.
Keep both letters in your pocket. V prices situations; Q prices choices. Every algorithm in this series is built from one of these two numbers — and you now know both.
Why World Models Especially Need Value: Seeing Past the Horizon
Here is the reason this tool matters specifically to world models, and not just to reinforcement learning in general.
A world model can imagine the future — but only so far before the dream gets unreliable. Small prediction errors compound, and after some number of steps the imagined trajectory drifts from anything real. Say the model is trustworthy for 15 steps. What about everything after step 15?
Value is the telescope. Imagine 15 steps in detail, then let V(s₁₅) — the value of the final imagined state — summarise all the rest in one number. A detailed dream up close; a single learned estimate beyond.
Every dreaming agent works this way: a short imagined rollout, plus value at the end. It is how an agent with a 15-step imagination still plays for outcomes hundreds of steps away. Remember this picture — when we build imagination-trained agents later in the series, this is the reason the critic exists.
Tool 4: Two Ways to Learn Behaviour
We now have reward, return, and value. One question remains: how does an agent actually decide what to do? The field settled on two answers.
Way 1: score every action, take the best. Learn Q(s, a) for every action, then behave greedily — always pick the action with the biggest number. That is Q-learning, the family behind DQN and the 2013 Atari breakthrough. Where it shines: a handful of discrete choices — left, right, jump. Score each, compare, done. Where it breaks: continuous actions. A robot arm's motor command is a list of real numbers — infinitely many candidates. You cannot score them all and take a max.
Notice something curious about Way 1: there is no "policy" stored anywhere. Behaviour is just read off the score table, fresh, at every step.
Way 2: keep an actor, and a critic. Split the job into two learners. The actor is a policy — a network that directly outputs the action: see the situation, propose the move. No table, no max — it can output continuous values as easily as button presses. The critic is a value function — a second network that watches and scores: "that choice raised your prospects," "that one lowered them." The critic's judgement is the actor's training signal.
The rhythm is: rehearse a move, get scored, adjust, repeat — a performer and a coach. The actor slowly bends toward whatever the critic rates highly.
Why World Models Chose Actor-Critic
World-model systems overwhelmingly use Way 2, and the reasons are structural, not fashionable.
First, dreams are perfect training grounds for a critic. Inside imagination, the agent can rehearse thousands of episodes cheaply — and the critic can score every imagined step. The actor-critic pair thrives exactly where a world model lives.
Second, real bodies have continuous actions. Robot joints, steering angles, torques. The actor outputs them directly; a Q-table cannot.
Third — and this closes the loop with Tool 3 — the critic is the horizon telescope. The short-dream-plus-value-at-the-end picture from the last section? The critic is that value. Actor, critic, and world model click together like parts of one machine.
This is why Dreamer-family systems — including the one we will reproduce as our capstone — are all actor-critic inside a dream.
Four Tools to Keep
Let us close by putting the toolkit on the table, because everything that follows in this series is assembled from exactly these four pieces.
1. Compress, then predict. An autoencoder squeezes each frame into a latent; the world model predicts in that small space, and decodes only when it wants to look. This is the series in one sentence.
2. Judge actions by the return, not the reward. Total future reward, gently discounted by γ. The whole difficulty of learning behaviour is that credit arrives late.
3. Value is a price tag on a situation. V(s) for places, Q(s, a) for choices. Value fills the silence between rewards — and sees past the imagination horizon.
4. Actor-critic is the world model's partner. A policy that acts, a critic that scores — the pair that trains happily inside a dream, and handles continuous actions along the way.
The toolkit is complete. Next lecture, we use every tool at once — on the 2018 paper that named the field: Ha and Schmidhuber's "World Models." Three small networks — an eye, an imagination, and a hand — each built from a tool you now own; an agent trained entirely inside its own dream, with the real game unplugged; and the trick that stopped the policy from cheating the dream. See you there.