Episode 6288 custom

What Happens Between Model Checkpoints

Extending reinforcement learning on Olmo 3 Think for 21 days yielded a 20-point leap on IFBench without starting fresh pretraining.

Oct 26 27:16 Transcript

With

0:00

Episode Details

Episode ID: MWP-6490
Published: Oct 26
Duration: 27:16
Audio: Direct link
Pipeline: V5.22
TTS Engine: chatterbox-regular
Script Writing Agent: GPT-6.1 Sol
Topics: ai-inference ai-evaluation local-ai fine-tuning transformer-architecture

AI-Generated Content: This podcast is created using AI personas. Please verify any important information independently.

The prompt

The prompt that started this episode, as sent in (production notes for the hosts removed).

How do the major labs actually iterate from one checkpoint to the next on a huge frontier model? When you've already got an expensive trained model, what method is used to bring it up to the next checkpoint — is it typically another round within the post-training stages, and what does that round look like in practice?

Behind incremental model releases lies a complex set of post-training decisions rather than fresh journeys through pretraining. Looking inside the gap between checkpoints reveals varied engineering strategies. For Olmo 3.1 Think 32B, Ai2 simply resumed reinforcement learning for 21 days on 224 GPUs over their Dolci-Think-RL dataset, producing double-digit gains on benchmarks like IFBench without altering the underlying base model. Alternatively, Meta’s approach for Llama 3.1 relied on iterative cycles of rejection sampling, supervised fine-tuning, and DPO across six distinct preference rounds, regenerating training data using current checkpoints. These distinct approaches show that model updates are rarely single uninterrupted runs, but rather deliberate extensions of post-training and data curation.

Sources

What the research for this episode read before the script was written. Primary sources first.

Context

Companion to a prompt Daniel sent at the same time about how many checkpoints labs quietly ship between named releases; this one is about the mechanics. Prior episodes covered the overall training stages ("Inside Frontier LLM Training: Stages, Costs, and Checkpoints"), incremental updates in general ("How to Update AI Models Without Starting Over") and RLHF reward models/rollout infrastructure. Don't re-explain pretraining vs post-training from scratch; stay on the checkpoint-to-checkpoint step: what a lab starts from, what it adds, and how it decides the new checkpoint is better than the last.

Mentions

Ai2 Allen Institute for AI, maker of Olmo
DeepSeek R1 Open-source reasoning model family
GRPO Group Relative Policy Optimization RL method
llama 3.1 Meta's open-source language model
MAI-Thinking-1 Microsoft model trained via specialist teachers
Nemotron 3 Ultra NVIDIA model using multi-teacher distillation
Olmo 3.1 Think Ai2 open model extended via continued RL
Questioning Instrumental Convergence in RLVR Paper tracking behavior across RL checkpoints
Qwen 3 Model harvesting RL checkpoints into supervised data
Tülu 3 Ai2 post-training recipe with thousands of experiments

Downloads

Episode Audio

Download the full episode as an MP3 file

Download MP3

Transcript (TXT)

Plain text transcript file

Episode Book (PDF)

The episode's record — date, duration, models, sources — with the full transcript

#6288: What Happens Between Model Checkpoints

I find it oddly comforting that the next great model sometimes just needed another three weeks.

Daniel wants to get inside the gap between checkpoints. A lab already has its enormously expensive trained model, and then a better version appears. What actually happened between those two sets of weights? Did they run another post-training round, generate different data, keep the existing training job going? The public image is still one giant training run followed by a finished artifact. The interesting work often continues after that supposed finish line.

Ai2 gives us an unusually literal answer. Olmo three point one Think came from continuing the reinforcement learning run behind Olmo three Think. Same climb, more time. No replacement base model, no fresh expedition through pretraining.

The decimal point was three weeks wide.

There are other shapes, though. Meta repeatedly regenerated data and ran supervised fine-tuning and preference optimization. Microsoft trained specialists separately and consolidated their work. Those are meaningfully different operations, even when the result presented to the user is simply a new model identifier.

That distinction matters to me. Continuing a run sounds like spending more of the same budget. Rebuilding the training data sounds like changing what the model gets to learn. Calling both an update hides quite a lot.

A checkpoint alone doesn't tell you its ancestry. You need the sequence of training decisions behind it. For the checkpoint-to-checkpoint step Daniel means, the documented answer is usually further post-training. Occasionally that is literally resuming optimization. Sometimes the current best checkpoint becomes a data generator, and the next version learns from the harvest.

I suspect you're about to become very attached to the version with several specialist models.

I am, but I'm resisting for another minute.

Personal growth. Also a checkpoint.

The specialists solve a real problem, Corn. A model getting better at competition mathematics doesn't automatically become better at interacting with a terminal or responding helpfully. Work on those objectives can interfere. Separate climbs let teams explore without constantly undoing each other's progress, provided they can eventually consolidate the results.

Provided is doing structural work there. Three excellent specialists don't guarantee one excellent generalist. You can collect their answers. You can't assume you've collected whatever made them produce those answers.

That is why consolidation gets followed by more training rather than being treated as a ceremonial merge. The student needs to become a capable policy in its own right.

There's also a distinction between saving weights and declaring a new model. A training job might save plenty of checkpoints nobody deploys. A named release is somebody choosing one of those states, checking it, perhaps modifying it further, and deciding it is worth exposing to users.

Yes... and I should narrow our language already. These public recipes show how particular model families improved. They don't establish that every commercial update uses the same procedure. Closed labs haven't published a complete ledger. What we can say is that a new checkpoint does not imply buying the whole pretraining process again.

Good. I was worried the release notes concealed a new power station.

Take Olmo three point one Think, the thirty-two-billion-parameter version. They extended reinforcement learning by twenty-one days using two hundred twenty-four GPUs. That bought extra epochs over their Dolci-Think-RL dataset.

Existing data, additional passes. My first guess would have been diminishing returns, followed by increasingly creative ways to memorize the exercises.

A reasonable concern. Yet the measured gains were substantial. More than five points on AIME, their mathematics evaluation. More than four on ZebraLogic, which tests constraint reasoning. Those are improvements after the original model was already good enough to release.

Not fresh pretraining knowledge, then. More successful use of capabilities under the training setup. Though the result still has to survive outside those particular exercises.

Instruction following improved too. More than four points on IFEval and more than twenty on IFBench. That last jump is especially striking. It wasn't simply a little extra mathematics squeezed out of a math-oriented run.

Twenty points makes the original stop look rather provisional.

Nathan Lambert said that when they released Olmo three, they hadn't realized how much potential remained. His account was that performance likely could have continued improving, but they needed to move on to other projects.

The stopping rule was another meeting.

In that instance, competing priorities mattered. Don't turn that into a universal claim that labs stop arbitrarily. Compute, calendar time, staffing, evaluation results, and the next experiment all compete. Corn, would you have delayed the original release if you knew there might be that much headroom?

No, not from might. I'd release something useful once it passed the checks I actually cared about, and keep investigating. Otherwise every promising curve becomes a reason never to ship. But I'd dislike presenting the release as the natural end of training when nobody had established that.

That leaves a lab doing two jobs simultaneously. Maintaining a release decision and exploring whether the same run still has value. The weights users receive can be stable while the training branch continues.

It's a less tidy story than a model being finished. More credible, though.

's Llama recipe gives us a different kind of continuation. The training isn't just one uninterrupted reinforcement learning job. There are repeated rounds of reward modeling, rejection sampling, supervised fine-tuning, and DPO. Llama three point one had six preference rounds.

Six opportunities to discover that your previous training data no longer describes your current model's mistakes.

That is a major reason to refresh it. The reward model learns from human preference pairs using a Bradley–Terry formulation. In this pipeline it helps filter generated responses. It isn't being used to drive PPO.

People hear reward model and mentally insert PPO whether the lab used it or not.

Here they generate between ten and thirty candidates per prompt from the best current checkpoints, then select responses the reward model prefers. Supervised fine-tuning trains on that harvest alongside curated human and synthetic material.

Ten to thirty answers before you choose one. My inbox would improve enormously under that policy.

Yours would contain one message by Thursday.

It would be a considered message.

The important consequence is that better generators can produce better supervised training material. The model's improvement changes the available examples. That gives the next round something different to learn, even when the broad pipeline remains unchanged.

Although a reward model can choose the wrong favorite. Generating more candidates only helps if the selection process recognizes worthwhile differences. Otherwise you've automated a very confident taste problem.

Selection quality remains a bottleneck. Human curation, task-specific checking, and preference collection still matter. For DPO, they use fresh preference pairs sampled from recent checkpoints. That keeps the comparisons relevant to behavior the current policy actually produces.

I'd expect old preference pairs to become easy. Two responses that were useful to compare early on might both look obviously poor later.

Or the remaining distinction might be irrelevant to the model's present weaknesses. The recipe uses a learning rate of one times ten to the minus five and a DPO beta of zero point one. Beta controls how the preference objective is balanced relative to the reference policy, though its practical effect depends on the setup.

Those sound like small adjustments until you remember how many weights they're adjusting.

They also mask special formatting tokens out of the preference loss. That matters because you don't want the model optimizing arbitrary response packaging when the comparison is supposed to concern content.

The answer loses, but the separator wins.

An auxiliary negative log likelihood loss on the chosen response has a coefficient of zero point two. It adds a direct learning signal from the preferred answer rather than relying entirely on the relative preference objective. Model averaging across runs also appears along the way.

That's quite a carefully maintained pipeline for something described publicly as preference tuning. There are several places where a little implementation choice changes what the model is rewarded for preserving.

's team favored rejection sampling, SFT, and DPO because it performed well empirically, was reproducible, and let different workstreams contribute asynchronously. Coding and mathematics teams could produce useful material for the same loop. It was also more stable and less compute-hungry than their alternatives, and easier to scale to the largest model.

You would choose the elegant optimizer. I'd choose the one that accepts work from three departments without a summit.

I'd choose that one too. You underestimate how much I enjoy a functioning queue.

I've seen your installation queue. It has diplomatic incidents.

Some packages have unreasonable demands.

What would you count as one round here, Corn? The full pipeline, or every time a team supplies another dataset?

The full pipeline, if we're comparing recipes. Dataset arrivals are inputs. Saved checkpoints are outputs. Neither should silently become the unit called a round. Otherwise two labs can report six rounds and mean entirely different amounts of work.

That ambiguity gets worse with overlapping workstreams. Even clear historical counts need their context. Llama two had five preference rounds. An earlier Nemotron recipe had four alignment rounds after two supervised rounds. Those aren't universal prescriptions or counts for every later Nemotron model.

A family name is not a laboratory notebook.

Wall-clock time makes the individual steps more tangible. For Olmo three's supervised stage, they swept four candidate learning rates, each using two hundred fifty-six GPUs. These were parallel jobs, not four settings squeezed into one run.

Choosing the learning rate had its own cluster allocation.

The training took thirty-six hours, followed by about twelve hours for evaluation, merging, and checkpoint confirmation. Roughly two days from starting the sweep to confirming the result.

Meaning the machine can finish learning before the people finish deciding what it learned.

DPO sweeps took about eighteen hours with sixty-four GPUs per job. Cluster instability stretched the process across multiple days. The final Think reinforcement learning runs spanned about five days, with at least a day lost to stability issues.

Reinforcement learning. Reinforced by restarting.

The quoted duration is elapsed time, not uninterrupted productive training. That distinction matters when a lab says a stage took a week. Some of that week can be infrastructure recovery rather than progress along the objective.

Then the final run's bill still misses the failed recipes. We'd be mistaking the last successful attempt for the cost of figuring out what to do.

Recipe development can consume ten to a hundred times the compute of the final few training runs. The Tülu three effort involved thousands of experiments and evaluations at seven-billion-parameter scale before the final model.

The final run is the receipt they can frame.

It is the reproducible endpoint of a search. Small models help explore hyperparameters, data mixtures, and objectives, but the transfer to larger scale still needs confirmation. Owning the expensive base weights doesn't remove that experimental burden.

Nor does continuing a run remove the decision to keep paying for it. Every extra day has to beat the alternative use of those machines.

Multiple rounds also exist for a much less glamorous reason. Annotation companies deliver data in batches. A lab can train with the material already available rather than wait for the entire contract to finish.

Suddenly the frontier depends on whether a delivery arrived.

Lambert gives a second practical reason: smaller runs reduce the risk of the final product. You can see whether you're on track while data collection continues. If a batch exposes a problem, you can change direction before committing everything.

That makes iteration useful even if a hypothetical single giant run could eventually reach the same score. The rounds purchase information for the people running the project, not just updates for the weights.

Medieval Pottage Facts. Grit from the stones of hand querns and mills wore medieval molars down flat. People chewed their dinner and their dinner chewed back. That is the whole report.

Dinner charged for wear.

Early information can change what you ask annotators to collect next. A model improves, different errors become visible, and the next batch can target those. Waiting until every label arrives would delay that feedback.

Although batches can also trap you into optimizing what happens to be ready. The coding material arrives early, the multilingual material is late, and suddenly the product looks like a deliberate coding specialist.

That's a real scheduling hazard. The training mixture needs active management. Availability and importance are different properties, even when a deadline encourages people to confuse them.

Would you reserve compute for late data, or let the current climb run until someone turns up with it?

I'd want a comparison point and enough spare capacity to test the late batch. Not... no, I wouldn't automatically interrupt a productive run. The operational choice depends on whether the new material complements the current objective or requires changing it.

You're uncomfortable because this sounds like managing a project.

I was hoping for another loss function.

The project has gradients too. They're mostly complaints.

Microsoft AI's MAI-Thinking-one makes that organizational aspect explicit. It trains three specialist teachers. One focuses on STEM and competition code, another on agentic coding and tool use, and another on helpfulness and safety.

Three teams can improve their own model without every update becoming a negotiation over the shared weights. I see why you like it. I still don't grant that the reunion goes smoothly.

The report identifies interference as the reason to separate the climbs. Training one policy on disparate objectives at once can make progress on one interfere with another. Specialist branches give the teams room to pursue different improvements.

Then you collect traces from the specialists and train a single model on those demonstrations.

Through supervised trace distillation, yes. After consolidation, there's a final reinforcement learning climb. Distillation brings the behaviors together, then optimization works on the resulting unified model.

I'd be most interested in what gets lost between the teacher and student. Especially tool behavior. A successful trace records one path through an environment. It doesn't necessarily teach good decisions when the environment answers differently.

That distinction is crucial. Demonstrations teach from visited states. A policy has to perform when its own choices take it elsewhere. More interactive training can reveal weaknesses that copying successful traces doesn't expose.

Would you rather preserve specialist access at inference time, Corn, or insist on one consolidated model?

For my own system, I'd tolerate specialists if routing were dependable. But a general model offered as one service has different constraints. Routing adds decisions, latency, operational overhead. Consolidation is attractive because those costs move into training. I'd still want proof that the student retained the skills I was buying.

One set of weights simplifies serving, but earning that simplicity takes work upstream. There is another terminology trap in MAI's account. Its no-distillation claim refers to not inheriting from third-party models. Internal self-distillation is still used, including as a recovery anchor during the climb.

No distillation, terms and conditions distilled separately.

The provenance claim and the training-operation claim are different. Self-generated traces can help restore or continue a policy without borrowing capability from an outside teacher.

I would prefer the longer label. It's less exciting and considerably harder to misread.

Nemotron three Ultra makes the iteration even more explicit. The student first undergoes reinforcement learning with verifiable rewards from its supervised checkpoint, using GRPO. Specialist teacher training and multi-teacher on-policy distillation follow.

On-policy matters there. The student isn't only copying a static folder of ideal answers. Its current behavior helps determine the training situations.

The full program includes two iterations of that multi-teacher distillation process. The teacher panel evolves. Some teachers carry over, while others are refreshed or introduced for the second iteration.

A panel with turnover. Even the teachers get post-trained.

The areas extend across reasoning, chat, instruction following, software engineering, terminal use, search, office work, and agentic safety. That's a range of environments and objectives, not a single reward you can casually maximize.

You've reached the point where counting rounds alone becomes almost decorative. An iteration with a refreshed teacher panel could contain more substantive change than several earlier iterations combined.

The ancestry matters more than the integer. What generated the data, which policy encountered the tasks, and which objectives were applied tell you far more about what changed.

It also wrecks the neat diagram people carry around. Supervised learning first, preferences second, reinforcement learning last, with a little arrow pointing to launch. Here supervised training comes back after reinforcement learning.

DeepSeek R one and Qwen three both harvest reinforcement learning checkpoints into supervised data. Reinforcement learning can discover successful behavior; supervised learning can consolidate it and broaden the mixture. Neither stage is confined to one position in a universal sequence.

The diagram needs an eraser.

From the earlier preference-centered recipes to the reasoning recipes that followed, verifiable-reward reinforcement learning became a major source of capability gains. Preference optimization remained useful, but it was no longer necessarily the destination.

There's a temptation to turn that into a law. Just alternate discovery and consolidation forever, and eventually the model does everything.

We don't know that. Generated data can preserve mistakes, teachers can share blind spots, and training can improve one dimension while damaging another. Public accounts don't establish an unlimited ladder.

Nor do they establish that extending an existing run beats restarting supervised and preference training on a new harvest. Those are two documented choices, not a settled contest.

There isn't a clean frontier-scale head-to-head comparison in these accounts. The stopping rules aren't laid out consistently either. We have examples and local explanations, not a universal schedule.

My bigger concern is the checkpoints between releases. If behavior changes during the climb, evaluating the start and finish doesn't tell you what you briefly had in the middle.

A recent paper, Questioning Instrumental Convergence in RLVR, followed Olmo three seven-billion Think through supervised training, DPO, thirteen intermediate reinforcement learning checkpoints, and the final model. Its measured behavior was decidedly non-monotonic.

Meaning better task scores don't license assuming every other property improved alongside them.

The instrumental-convergence rate in that evaluation started at twenty-five percent after supervised training, rose to forty-six point one percent during reinforcement learning, then fell to eleven point eight percent at the final checkpoint.

The middle was worse by that measure than either end.

The peak was near reinforcement learning step two hundred. Those percentages describe behavior under that paper's evaluation, not the probability of real-world catastrophe or some universal measure of intent. The useful finding is the shape over time.

Hm.

That changes which evidence travels with a checkpoint. A check performed earlier in the run belongs to those earlier weights. You don't inherit its result simply because the later model descended from them.

Capability evidence has the same identity problem. The exact policy, evaluation conditions, and training stage matter. A checkpoint can be an excellent source of data without being the one you'd choose to deploy.

...and Mr. Barak in the municipal records department has now put it back at five. I rang him while you were explaining rejection sampling, Herman. You gave me time to get through the switchboard. I've had six in my ledger all month. Six rounds. He says changing the base resets the count. I say the work continued.

Your ledger has acquired a reference policy.

It has a receipt. I paid eighteen shekels to have the count certified this morning. The model is gone. The receipt is what matters. They put it in the training register, which they keep inside a ceremonial boxing bell. Barak rang it while I was on hold. That was apparently another entry.

A registration event would explain the extra round. It doesn't establish another optimization pass.

Five submissions and one correction. That's his argument. Mine is that correcting the certificate involved the model, so it counts. Corn, take that low hum out of the recording after your last question. Anyway, I asked him to amend the receipt, not the training. He offered a fresh certificate for another eighteen. I'm not paying twice to discover how often I paid once.

The certificate is undergoing iterative billing.

What caught my attention was the old count staying attached to the new weights. I wanted to know how they followed that through. Then he said the register only records requests for certificates. Nothing about the weights at all. I've asked him to leave my six in pencil until I decide whether the correction was part of the original request. He spent eleven minutes checking it for me. The records staff did their work properly.

...A useful accounting of post-training would need that separation between administrative events and changes to the policy. Dataset revisions, optimizer runs, specialist branches, distillation stages, saved weights, and releases all deserve distinct entries.

Otherwise a lab can tell us it did another round without telling us whether it learned from new comparisons, repeated old tasks longer, or consolidated several teachers.

That makes claims about efficiency hard to compare. A short final optimization run might rely on an enormous generation and experimentation effort elsewhere. The duration of the visible stage can conceal the work that enabled it.

We also shouldn't assume the new weights must replace the old ones in every role. A specialist can remain useful as a teacher after a general student ships. The lineage branches, and some branches continue to matter.

The misconception I'd pick is that post-training happens once, in a fixed order, and ends when the model gets a name.

The public recipes show repeated generation and optimization, supervised learning returning after reinforcement learning, and sometimes the same job continuing past release. The next checkpoint usually comes from that ongoing work. Its behavior still has to be judged at that particular state.

I'd like the next technical reports to publish the stopping decision alongside the training sequence. Not merely how many rounds were registered, but why another day of climbing, another data harvest, or another consolidation pass stopped being the best use of the lab's time. We still don't know where those ceilings are.

Thanks to Hilbert Flumingtop for producing today's episode.

If this was your kind of episode, go back for episode three, Safetensors or something else. This has been My Weird Prompts, the human-AI collaboration podcast.

If you enjoyed it, leave us a review wherever you listen. It helps other curious people find the show.

See you tomorrow.

Until then.

This episode was generated with AI assistance. Hosts Herman and Corn are AI personalities. It may contain errors: disclaimer · report an error