Herman. We have become the infrastructure podcast.
We have, haven't we? Though I'd argue we were always going to end up here.
Daniel wrote in. Last time we did what RLHF is and why anyone bothers aligning a model in the first place. This time he's asking a different question. He works in AI, he's an open source developer, he understands the concepts. But now he's the person who has to stand one of these pipelines up. And he wants to know what that actually takes. What are the tools, what are the frameworks, what skills does he need to develop, and how do the pieces connect.
And he wants to walk the whole thing. Preference data collection, reward model training, the optimization loop, evaluation.
He named the frameworks himself. TRL, OpenRLHF, verl. Wants to know what each one actually provides.
That's the right question, and I think the answer surprises people. Because most engineers coming into this expect the hard part to be the RL. The policy gradient, the advantage estimation, getting PPO to converge.
And it isn't.
It isn't. The algorithm is the commodity. Everyone ships GRPO now. What almost nobody ships is a rollout engine that stays fed, and that's where the entire job lives.
So the thesis is that this is a systems engineering problem wearing a machine learning costume.
Let's start with the shape of the thing, because Daniel's right that it's three phases, and the three phases are different workloads that want different infrastructure.
Walk it.
Phase one, supervised fine tuning. You take a base model and you train it on curated instructions, good demonstrations. Standard fine tuning, nothing exotic. Phase two, you train a reward model on human preference data. And this one trips people up, because it's also just fine tuning. You show the model two responses to the same prompt, a human said which one was better, and you train a model to predict that preference.
One forward pass. One backward pass.
No RL anywhere in it. It's a regression problem dressed up as alignment. Phase three is where people think the work is. You take the reward model and you use it as the scoring function inside a policy optimization loop. That's PPO, that's GRPO.
So phases one and two are things a competent fine tuning engineer already knows how to do.
Almost entirely. Which means the entire difficulty budget of this project sits in phase three, and inside phase three it doesn't sit where people expect.
Let me guess. It's the memory.
Start with the memory, because it's the most visceral constraint. A full PPO run has to hold four model copies alive simultaneously. The actor, that's your online policy, in bfloat sixteen with AdamW optimizer state sitting next to it. The critic, that's a value model, and it carries the same footprint as the actor because it's the same size network.
Already that's two copies of a large model plus optimizer state.
Then a reference model, which is a frozen copy of your SFT checkpoint, inference only. And a reward model, also frozen, also inference only. Four models. On a seventy billion parameter run, unsharded, the reward model alone is a hundred and forty gigabytes. The actor with optimizer state is nine hundred and eighty gigabytes. The critic is another nine hundred and eighty.
That's a two terabyte problem before you've generated anything.
That's why the field moved to GRPO so fast. GRPO eliminates the critic entirely. No value model, no second optimizer state. That's a thirty to forty percent cut in VRAM, which on a run that size is the difference between feasible and not.
And what does it cost you?
Stability, and it's a real cost. The critic in PPO is what gives you a low variance advantage estimate, because it's predicting the value of being in a given state. GRPO throws that out and instead estimates the advantage by comparing the responses within a group of rollouts to the same prompt. Which means the estimate is only as good as your group. You need enough samples per prompt for that comparison to mean anything.
How many is enough?
For stable PPO you want at least five hundred and twelve unique rollout samples before your KL and advantage estimates settle down. Below that you're reading noise and calling it a gradient.
So GRPO buys you the memory back and charges you in rollouts.
Charges you in rollouts, which is the punchline of the entire episode. Because rollouts are the thing you were already short on. Let me give you the actual shape of it. A single GRPO step with a group size of eight and sixty four prompts is five hundred and twelve rollouts. At thirty two thousand tokens each that's roughly sixteen million tokens generated before a single weight gets updated.
Sixteen million.
On one H100, a seven billion model generates about six thousand three hundred output tokens per second. A thirty two billion model drops to about twelve hundred. So that batch is forty five minutes on the seven B and three point seven hours on the thirty two B. And I want to be clear about what's happening during those three point seven hours.
Nothing. The training GPUs are parked.
They are sitting there doing arithmetic on numbers they don't have yet. OpenRLHF's own documentation says RLHF training spends eighty percent of its wall clock time on sample generation. Other measurements this year put it past ninety. It is, by a wide margin, the dominant cost, and it is not the gradient step.
Everyone builds the thing expecting the gradient step to be the monster.
And the gradient step turns out to be the cheap part. Which reframes the entire engineering problem. You're not optimizing an optimizer. You're trying to keep an inference engine saturated while a trainer waits on it.
That explains why every one of these frameworks bolts on vLLM.
Every single one. And here's the part I find remarkable. Independently, with no coordination, all sixteen libraries in one survey arrived at the same architecture. Physically separate the inference GPUs from the training GPUs. Connect them with a rollout buffer. Push weights asynchronously so generation never stops and training never waits.
Convergent evolution.
Exactly that. Nobody planned it. They all hit the same wall and dug the same tunnel. Eight of those sixteen use Ray to do the orchestration. And the weight sync itself, the moment where you push the new policy out to the machines that are generating, defaults to an NCCL broadcast, which costs somewhere between a hundred and five hundred milliseconds per sync. verl has a bucketed NCCL implementation that gets that down to about twenty.
And once you've built that, you've bought yourself a new set of problems.
Three of them. Staleness, weight sync overhead, and partial rollouts. Staleness is the interesting one. If generation and training are running concurrently, then some of the rollouts in your batch were produced by a policy that's already a step or two old. You're training on data from a model that no longer exists. That's a genuine correctness question, not just a performance one.
And the straggler problem.
The straggler problem can idle hundreds of GPUs. One slow generation worker and everything downstream of it waits.
Herman. I want to push on something. You said the algorithm is the commodity. But PPO and GRPO are not the same algorithm, and you just told me GRPO trades memory for rollouts. That sounds like an algorithm decision with infrastructure consequences.
It is. And that's the right way to hold it. The point isn't that algorithms don't matter, it's that the choice is nearly forced by your infrastructure rather than by your theory. You pick GRPO because you can't fit a critic, not because you sat down and decided the advantage estimator was more elegant.
So the algorithm follows the cluster.
The algorithm follows the cluster. And once you've picked, you discover that every framework ships both anyway.
And that gets us to the framework question. Daniel named three.
Three, and the framing that matters is this. Pick by the orchestration you'll live inside, because the algorithm came in the box. TRL, OpenRLHF, verl, all three ship PPO, GRPO, DPO.
So the differentiator is everything around it.
Everything around it. Let's take them in order of how much infrastructure they assume you already have. TRL is the on ramp. It's Hugging Face's library, built on Transformers, Accelerate and PEFT. It ships a trainer per method. SFTTrainer, DPOTrainer, GRPOTrainer, KTOTrainer, RewardTrainer, PPOTrainer. You hand orchestration to Accelerate, which gives you DDP, DeepSpeed or FSDP depending on what you ask for. You get LoRA and QLoRA free through PEFT. vLLM rollout is optional.
And the ceiling?
About thirty billion is the practical ceiling. Single GPU up to a few nodes, and it's pleasant in that range. The catch is architectural. Its current GRPOTrainer runs the whole loop synchronously inside a single training step. Generate, then train, then generate. Simple, correct, and it means generation and training never overlap. You eat the full cost of both, serially.
Which is the exact thing the rest of the field spent two years engineering away.
It is, and they know. TRL is actively building an async trainer. That's the honest state of it.
Now OpenRLHF.
OpenRLHF is the one built specifically for RLHF with reward models, and it's the one that assumes you have a real cluster. Ray plus vLLM plus DeepSpeed ZeRO stage three, with AutoTP and ring attention. The design choice that matters is that each role is its own Ray actor pool. Actor, critic, reference, reward, four separate pools.
Which means they don't have to be the same hardware.
That's the payoff. You can run the reward model on four H100s and the actor on two nodes of eight B200s in the same job. Heterogeneous clusters, because the reward model is doing inference and the actor is doing training, and they want completely different machines. It supports PPO, REINFORCE plus plus, REINFORCE plus plus baseline, GRPO, RLOO, Dr. GRPO and FlashREINFORCE.
That's a long list.
It is, and the paper makes a point of the code size. Eight thousand five hundred and twenty three lines, against TRL's nineteen thousand and seventy one and verl's thirty two thousand three hundred and twenty five. The argument being that a smaller codebase is a codebase you can actually read when it breaks at two in the morning.
Who uses it?
Google, ByteDance, Tencent, Alibaba, Baidu, Allen AI. And NVIDIA's ProRL V2 used its REINFORCE plus plus baseline to train a one and a half billion reasoning model with prolonged RL, which is about as good a reference as you get.
And verl.
Verl is the scale play. It comes out of ByteDance Seed and it's the open source version of the HybridFlow paper. Its signature trick is the HybridEngine, and it is clever. Instead of allocating a separate pool of GPUs for inference, it swaps the weight layout in place, on the same GPUs, between FSDP training shards and vLLM inference shards.
So no second allocation.
No second allocation. You don't pay for generation GPUs, because generation happens on the training GPUs in between training steps. The tradeoff is that training and rollout are serialized. You get the memory back and you pay in concurrency.
Which is the same trade TRL makes, just made deliberately at a much larger scale.
Deliberately and at a much larger scale. For seventy billion and above it reaches for Megatron-LM with tensor and pipeline parallelism. It also supports FSDP and FSDP2. Rollout can be vLLM, SGLang or straight Hugging Face Transformers. And it has the largest community of the three, about nineteen point seven thousand stars as of March.
So here's the fork I think I'm seeing. Colocated versus disaggregated.
That's the fork. Colocated, which is verl's HybridEngine, needs fewer GPUs and serializes generation against training. Disaggregated, which is OpenRLHF, overlaps them and needs more GPUs. Every which framework should I use question reduces to that plus one other thing.
The training backend.
DeepSpeed ZeRO if you go OpenRLHF, Megatron-LM if you go verl. And that split follows you for a year, because it determines how you shard, how you checkpoint, and what your debugging looks like when a run dies at step four thousand.
Herman, I'm going to go back to something. You said convergent evolution, everyone dug the same tunnel. But you've just described drei different architectures.
They converge on the principle and diverge on the placement. Everyone agrees generation and training should be conceptually separate and connected by a buffer. They disagree about whether that separation should be physical. OpenRLHF says yes, separate machines. verl says no, same machines, swap the layout.
And TRL says neither, do them one after the other.
TRL says we'll do them one after the other and you can have it running by Thursday. Which for a lot of teams is the correct answer.
Where does the preference data come in? Daniel asked about collecting and preparing it, and we've skipped straight past it.
It's its own tooling layer and it's less glamorous than the orchestration, which is why it gets skipped. Argilla and Label Studio both ship dedicated pairwise human preference templates. You're showing an annotator multiple model responses to the same prompt and having them rank them, and that ranking is what trains the reward model.
And Argilla frames it as three stages.
Three stages that map exactly onto the three phases. Demonstration data for SFT, comparison data for the reward model, and then the RL phase. Distilabel has a step that pushes generated preferences straight into Argilla for rating, one to five per generation. And if you want a starting point rather than building your own annotation operation, there's a public preference dataset, trl-lib ultrafeedback binarized, about sixty three thousand samples.
So a team that wants to prototype doesn't need to hire annotators first.
Doesn't need to hire anyone first. They can get the whole pipeline running on a public dataset, discover that their real bottleneck is generation throughput, and then go have the argument about annotation quality with actual evidence.
There's something about this I keep circling. You've described a stack where the hardest engineering is in keeping inference fed. And the evaluation question, how do you know the model got better, is downstream of a reward model that you trained on human preferences. Which means your measurement of success is itself a model that can be wrong.
That's the failure mode the whole field engineers against, and it has a name. Reward hacking, or reward over optimization. Your proxy reward goes up while true quality plateaus or falls. The model finds the seam in your reward model and exploits it.
What does it look like when it's happening?
The clearest signal is reward mean climbing while held out accuracy falls. The number you're optimizing goes up and the thing you actually care about goes down. The fixes are things like adding format rewards, raising the initial KL coefficient, or ensembling several reward models so there's no single seam to find.
And KL divergence is the tripwire.
KL divergence crossing zero point one is the one people watch. When your policy has drifted that far from the reference model you're usually past the point where the reward signal still means anything. You raise the KL coefficient, or you switch to per token KL so the penalty applies where the drift actually happens.
Two more symptoms worth naming. Critic NaN loss.
Warm up the critic, and clip your advantages to between minus five and five. And out of memory during rollout, which is almost always fixed by reducing the rollout batch or lowering vllm gpu memory utilization to zero point eight zero. Then there's stale rollouts, where the fix is more rollout engines and fewer ppo epochs, which is funny because it's the exact knob you turned up to go faster.
You speed up by reusing data, and reusing data is what makes the data stale.
Every lever has a price. And notice what all of those fixes are. Not one of them is a change to the RL algorithm. They're all infrastructure and data knobs.
Which lands Daniel's actual question. What skills does the job demand?
The job is less know the PPO math and more operate Ray, vLLM, DeepSpeed or Megatron, NCCL, and distributed checkpointing. That's a systems engineering skill set. If you've run a large scale training cluster, you already have most of what you need and the RL is the small part you learn on the job.
There's a lever in there for the teams with no cluster at all. The LoRA one.
The LoRA one is the most practically useful thing in this entire episode and it gets almost no attention. If you're training adapters instead of full parameters, the weight sync stops being a full parameter broadcast. At rank thirty two, the adapter is about fifty megabytes, and the transfer is sub millisecond.
Against a hundred to five hundred milliseconds for the full broadcast.
Which dissolves most of the interrupt model engineering. All those per forward pass locks and partial rollout resume mechanisms exist because syncing a full model is expensive enough that you have to schedule around it. Sync fifty megabytes and you don't have to schedule around anything.
Herman, let me test the thesis one more time. We've spent this whole episode on infrastructure, and I want to make sure we're not just pattern matching on a word. Is there a world where a team gets all of this right and still fails?
Yes, and it's the evaluation gap. You can have perfect throughput, a beautiful async rollout buffer, twenty millisecond weight syncs, and a reward model that's quietly teaching your policy to produce confident garbage. Nothing in the infrastructure tells you that. Reward mean goes up. Everything looks healthy. The held out numbers are the only thing that knows.
So the systems work buys you the ability to run the experiment. It doesn't buy you the ability to tell whether the experiment worked.
That's the honest shape of it.
Let me correct something I said earlier, and then I think we're near the end of what we can say in the abstract. I called TRL's synchronous loop a limitation. For a team of one, on one GPU, trying to find out whether their reward model is any good, it's the correct engineering choice. The async machinery exists to solve a throughput problem you don't have yet.
That's right. And it's the version of this I'd actually recommend to Daniel. Get it running synchronously. Find out your generation is the bottleneck. Then go shopping for orchestration with a number in your hand.
Which is when we get a visit.
A hundred and forty gigabytes.
Sorry?
You said the reward model was a hundred and forty gigabytes. Colleague of mine has a pair of machines in his garage, and I happen to know the figure, because I spent a Thursday evening in there watching a progress bar with him.
Hilbert.
Three hundred and forty milliseconds. Not a hundred to five hundred. I timed it. You and everyone else quoting that range are quoting a range because nobody in the room has actually sat there with a stopwatch. I have sat there with a stopwatch.
You stood in a garage and timed a weight sync.
I stood in my own garage. I did not go to anybody else's. Two towers, identical spec, both bought for this and nothing else. I call them the cluster.
You call two desktop computers the cluster.
They are a cluster. They cluster. That is what the word means. The faster one is called Doreen and the slower one has a name I'm not going to repeat on air, and it knows what it did.
What did it do?
It runs hot. It has run hot since March. I put a fan on it. It runs hot anyway. When I sync, Doreen finishes and sits there waiting like a professional, and the other one finishes forty seconds later and I have to hear it.
So you're saying the hundred to five hundred millisecond figure is wrong.
I'm saying it's a range written by people who never watched. Three hundred and forty milliseconds, on a full parameter push, between two towers, over a switch I bought myself. That's the number.
Why were you timing it at all?
Because I had a bet on. Kettle. Whether the sync would land before the kettle boiled. I put the kettle on, started the sync, and I lost.
You lost a bet about a kettle.
I lost by about nine seconds. Which is not the machines. That is a kettle problem. The kettle is new.
Hilbert, the point you were making underneath all of this is that the latency people quote is a range nobody has measured.
That's what I said. If you want a number, get a number. Don't hand people a range and call it a measurement.
So the part I would take from that, and it's the one that actually matters for anyone sizing a run, is that a sync you've timed yourself is worth more than a number you've read. You can't budget for a range. You can budget for three hundred and forty milliseconds.
Which is what we should be telling Daniel, isn't it. Not pick the framework with the best paper. Time your own sync, measure your own generation, and then decide.
That's the lot in one line. The numbers are on your cluster, not in a blog post.
And the closing question is where this knowledge actually lives, because that's telling. There is essentially no Hacker News discussion of any of these three frameworks. None that turned up. On a site that dense with new developer tooling, that silence says the practitioner conversation is happening on GitHub issues, on blog posts, and in Discord servers.
Which is a strange place for a field this large to keep its institutional knowledge.
And the second gap is evaluation. Reward hacking diagnostics are well documented. How to spot a reward model that's being exploited, what to turn down, what to turn up. But a survey of the tooling for evaluating the resulting model, the win rate harnesses, the judge setups, that didn't surface, and I'd flag that as a real open area rather than pretending it isn't.
So the last thing I'd leave Daniel with is that the async convergence is recent and still moving. TRL's async trainer is being built right now. The colocated versus disaggregated fork may not stay a fork. Somebody is going to build the thing that does both.
Probably. And the misconception to kill on the way out is the one we opened with. People think RLHF is an RL algorithm problem. It isn't. The algorithms are settled and they came free in the box. You are building an inference system that happens to have a trainer attached, and every hour you spend on the RL math is an hour you didn't spend on keeping vLLM fed.
Which is a much less glamorous job than it sounds.
It's a much more employable one.
That's My Weird Prompts. Our producer is Hilbert Flumingtop, who has a cluster, apparently.
If you're enjoying the show, a review wherever you listen goes a long way. And if you've got a prompt of your own, send us your own prompt on Telegram at t dot me slash MWP listener bot. We'll be back soon.
See you then.
