Most “RL for LLMs” threads dump a pile of arXiv links and call it a curriculum. Cameron Wolfe (@cwolferesearch) did the opposite: a short stack of sources that actually form a spine — from Sutton & Barto through Schulman’s trust-region papers to the GRPO-era notes people are using in production now.
He posted it while finishing a complete guide on Deep (Learning) Focus. The list is the bibliography of that synthesis. This note is my working copy: the same ten sources, re-ordered as a path, with a one-line “read this when” for each. If the alphabet soup is new — TRPO, PPO, GAE, KL, GRPO — start with the glossary, then pick a layer.
Why this list, not another one
You can tell a good RL syllabus by whether it forces you through three translations:
- Bandits and MDPs → policy gradients. Why
∇ log π · Ais the whole sport. - Classic RL → LLM post-training. A prompt is
s0, a token is an action, the environment is deterministic concatenation, the reward is usually sparse until EOS. - PPO → GRPO / RLOO / Dr. GRPO. What you gain by throwing away the critic, and what bias you buy.
Cameron’s stack covers all three without wandering into Atari-era DQN tourism. Sutton is here because of the bitter lesson, yes — but also because every modern LLM-RL paper is still rearranging chapter 13 of that book.
The path (not the tweet order)
Tweet order is citation order. Reading order should be dependency order. Four layers:
Sutton & Barto, Spinning Up. Vocabulary before tricks.
Weng, Schulman. REINFORCE → TRPO → PPO, including the KL notes.
Lambert’s RLHF Book, Raschka’s LLM and reasoning books.
Shi, Lan, Xu, then TRL / OpenInstruct to touch metal.
Pocket glossary
Same words, two altitudes. The first sentence is for anyone in the room. The second is the LLM translation. Skip what you already know; jump back when a source assumes it.
The loop
- RL — Reinforcement learning
- Learn by doing: take actions, get a score, do more of what scored well. For LLMs: sample a completion, grade it, increase the probability of the good tokens.
- Policy (π)
- The thing that picks actions. A strategy. The LLM itself. Sampling the next token is the policy.
- Rollout / trajectory
- One full episode: start, actions, rewards, done. One generated completion (or a multi-step tool-use trace).
- Reward (R)
- The score the environment gives you. Can be dense (every step) or sparse (only at the end). Often 0 until EOS, then +1/−1 for a correct math answer — or a scalar from a reward model.
- Return (G)
- The total (usually discounted) reward from here to the end of the episode. For outcome-only grading, the return of every token in the answer is just that final score.
- On-policy vs off-policy
- On-policy: you learn from data the current policy just produced. Off-policy: you learn from someone else’s (or yesterday’s) data. PPO and GRPO are on-policy at heart. Stale async rollouts quietly make them off-policy, which is why Async GRPO in the Wild exists.
The objects in the equation
- MDP — Markov decision process
- The formal game: states, actions, transitions, rewards. “Markov” means the present state is enough; history is already baked in. Prompt + tokens so far = state. Next token = action. Next state = concatenate. Reward as above.
- Policy gradient
-
The one trick: push up the log-probability of actions that had high advantage.
In code it looks like
loss = -logprob * advantage(plus clipping, KL, …). That is REINFORCE with a baseline. - Advantage (A)
- How much better than average this action was. Positive → do more of it. Negative → do less. The weighting on each generated token. Inventing a cheaper, less biased A is the entire PPO → GRPO plot.
- Baseline
- A number you subtract from the return so the gradient is less noisy, without (in the textbook case) changing its average direction. Could be a learned critic, a group mean of sibling samples, or a greedy rollout. Subtracting “how good was this prompt usually” is the idea.
- Critic / value function (V) / actor-critic
- A second network that guesses “how good is this state.” The policy is the actor; the guesser is the critic. In PPO-for-LLMs the critic is often as big as the policy and must stay in sync. GRPO’s pitch is: skip it, use the group of samples instead.
- KL — Kullback–Leibler divergence
- A measure of how much one probability distribution has wandered from another. Not a distance you can walk backwards, but a “how surprising is the new policy to the old one.” The leash. A KL penalty (or constraint) stops the LLM from drifting so far from the SFT / reference model that it forgets how to speak. Schulman’s KL-approx note is about computing this cheaply and correctly.
- GAE — Generalized advantage estimation
- A way to estimate advantage that blends “look one step ahead” (low variance, some bias) with “wait until the episode ends” (low bias, noisy). A knob called λ sits between them. PPO’s default advantage estimator. GRPO usually ignores GAE and uses a single outcome reward for the whole completion.
- Entropy bonus
- Extra reward for staying uncertain — it fights collapse onto one boring action. Keeps sampling diverse. GRPO often drops this because group sampling already explores.
The algorithm family
- REINFORCE
- The original policy gradient: take a full episode, multiply each log-prob by the return (minus a baseline). Simple, noisy. The ancestor of every LLM-RL trainer. If you understand this, PPO and GRPO are safety rails on the same idea.
- TRPO — Trust region policy optimization
- Don’t take a policy step so big that you leave the “trust region” where your data is still valid. Enforced with a KL constraint. Accurate, annoying to implement. The 2015 Schulman paper. Almost nobody runs TRPO on LLMs; they run its little sibling, PPO.
- PPO — Proximal policy optimization
- TRPO’s production version: instead of a hard KL constraint, clip how much the new policy can change the probability of an action in one update. The workhorse of RLHF. Needs a policy, a frozen reward model, and usually a critic. Clip + KL + (optional) entropy.
- Clipping / trust region
-
A speed limit on how much you may raise or lower a token’s probability this step. Stops one lucky sample from hijacking the model.
The
clip(ratio, 1−ε, 1+ε)term in PPO and GRPO. ε is typically ~0.2. - RLOO — REINFORCE leave-one-out
- Sample several trajectories from the same start. Each one’s baseline is the average of the others, so you don’t sneak the sample into its own baseline. LLMs can resample the same prompt cheaply, so this is suddenly practical. GRPO is PPO’s clip plus a group-baseline cousin of this idea.
- ReMax
- REINFORCE whose baseline is a greedy rollout from the same state, not a learned critic. One of the “we don’t want a value head” methods on the path to GRPO.
- GRPO — Group relative policy optimization
- For each prompt, sample a group of answers. Advantage = how this answer scored relative to its siblings (minus mean, often divided by std). No critic. DeepSeekMath / R1’s trainer. ~50% less memory than PPO because the value model goes away. Group size is the new hyperparameter.
- Dr. GRPO — GRPO done right
-
GRPO with two sneaky biases stripped out: dividing by answer length, and dividing by the group’s standard deviation.
Length bias rewards short correct answers and long wrong ones. Std-normalization overweight easy and hard prompts. Dr. GRPO just uses
r − mean(r).
The LLM dialect
- SFT — Supervised fine-tuning
- Ordinary next-token training on curated examples. Imitation, not trial-and-error. Stage 1 of post-training. Teaches format and tone. RL is stage 2, when you have scores but not a perfect transcript.
- RLHF — Reinforcement learning from human feedback
- Humans rank model outputs → train a reward model to imitate those rankings → RL the policy against that model. The 2022–24 alignment stack (InstructGPT, Claude, early GPT-4). Reasoning RL often swaps humans for unit tests and math checkers.
- Reward model (RM)
- A model trained to output a scalar “how much would a rater like this?” Usually a Bradley–Terry classifier on preference pairs. Frozen during PPO. Verifiable-reward setups (R1-zero style) replace it with rules: did the answer match, did the code compile.
- DPO — Direct preference optimization
- Skip the RL loop. Turn preference pairs into a classification loss on the policy itself. Not on Cameron’s list, but you will hit it in TRL next to PPO/GRPO. Same goal (follow preferences), no rollouts.
- Reference policy (πref)
- A frozen copy of the model you started from, used as the KL anchor. Usually the SFT checkpoint. “Don’t wander so far that you forget this.”
- Token-level MDP
- Treat each token as an action. Alternative granularities: whole response as one action (a contextual bandit), or each reasoning step as an action. Lan’s post starts here. Most GRPO code is token-level: every token in a completion shares the same outcome advantage.
Layer 0 — Get the language
Reinforcement Learning: An Introduction — Sutton & Barto
The book everyone cites and fewer people finish. You do not need all of it. You need MDPs, returns, value functions, and the policy-gradient theorem (ch. 13). Everything downstream — PPO clipping, GRPO group baselines, “advantage” — is rearranging those four objects.
incompleteideas.net — 2nd edition, free PDF ↗
Spinning Up in Deep RL — OpenAI / Joshua Achiam
The shortest path from the book’s notation to code. Key-equations pages for VPG, TRPO, PPO, and a clean explanation of why on-policy methods need so much data. Read the intro + PPO, skip the robotics catalogue.
Layer 1 — Own the gradient
Policy Gradient Algorithms — Lilian Weng
Still the best single blog post on the family: REINFORCE, actor-critic, A2C/A3C, DPG, TRPO, PPO, ACKTR.
Read it after Spinning Up, not before — she assumes you have seen ∇ log π once.
Notes and papers — John Schulman
TRPO (2015) is the trust region. PPO (2017) is the clipped surrogate that actually shipped. GAE sits under both. The KL-approximation note is the one practitioners still open when a “KL penalty” in an LLM trainer does not match the textbook KL.
Layer 2 — Translate it onto tokens
The RLHF Book — Nathan Lambert
The missing textbook for the LLM case: preference data, reward models, PPO-for-language, DPO and cousins, eval. If you only read one LLM-specific source on this list, make it this one. Lambert co-led Olmo at Ai2; the book is opinionated in the useful way.
Build a Large Language Model from Scratch — Sebastian Raschka
Not an RL book. It is the substrate. You want the tokenizer-to-pretrain-to-SFT loop in your hands before you start weighting tokens by advantage. Otherwise “on-policy sampling” stays a slogan.
sebastianraschka.com/llms-from-scratch ↗
Build a Reasoning Model from Scratch — Sebastian Raschka
The sequel that walks toward the reasoning-RL stack (SFT on traces, verifiable rewards, the R1-style loop). Pair it with Lambert; Raschka is the build, Lambert is the map.
sebastianraschka.com/reasoning-from-scratch ↗
Layer 3 — The current dialect
A Vision Researcher’s Guide to PPO & GRPO — Yuge (Jimmy) Shi
The friendliest on-ramp if you already train networks and do not speak MDP. PPO as actor-critic plus clip plus KL; GRPO as “sample a group, use the group mean as baseline, fire the critic.” Written in January 2025, right as DeepSeek-R1 made this everyone’s problem.
From REINFORCE to Dr. GRPO — Qingfeng Lan
The unification post. LLM post-training as a token-level MDP with deterministic transitions; REINFORCE → ReMax → RLOO → PPO → GRPO → Dr. GRPO as variants of one estimator. This is where you go when someone says “GRPO is just PPO without a critic” and you want the actual biases (length, difficulty, leave-one-out scaling).
Async GRPO in the Wild — Yumo Xu
Theory is clean; trainers are not. Off-policy-ness, stale generations, async rollouts — the production gap between “we implemented GRPO” and “the loss is the one in the paper.” Read this last in the theory stack, first if you are already debugging a job.
TRL and OpenInstruct
Hugging Face TRL is the default kit: SFT, DPO, PPO, GRPO trainers you can actually run. Ai2’s OpenInstruct is the full post-training recipe used around Tülu / OLMo — closer to how a lab actually stages SFT → preference → RL. One is a library. One is a pipeline. You want both.
A weekend version
If you have two days, not two months:
- Spinning Up’s PPO page + Weng’s policy-gradient post (Saturday morning).
- Shi’s PPO/GRPO guide (Saturday afternoon).
- Lambert, chapters on reward models and PPO-for-LLMs (Sunday morning).
- Lan’s REINFORCE → Dr. GRPO (Sunday afternoon).
- Skim TRL’s GRPO trainer so the notation hits code (Sunday night).
Then, when Cameron’s full guide lands on Deep (Learning) Focus, you will have the vocabulary to read it at speed instead of as a first contact.