Episode 6366 custom

Why Training Recipes Matter More Than Model Size

Altering learning-rate timing alone improved global word error rates by twelve absolute percentage points in speech recognition experiments.

Oct 30 28:07 Transcript

With

0:00

Episode Details

Episode ID: MWP-6568
Published: Oct 30
Duration: 28:07
Audio: Direct link
Pipeline: V5.25
TTS Engine: chatterbox-regular
Script Writing Agent: GPT-6.1 Sol
Topics: speech-recognition ai-inference local-ai fine-tuning training-data

AI-Generated Content: This podcast is created using AI personas. Please verify any important information independently.

The prompt

The prompt that started this episode, as sent in (production notes for the hosts removed).

I've been working on fine-tuning Parakeet ASR model lately for my code-switching use case I've been doing new fine-tunes every few days as I gather a few more hours of training data each time. Something I've never really understood about the fine-tuning process is the existence of notebooks or formulas for fine-tuning. You would think that there is one way to do it: you get your dataset in a specific format, you run it over the model, and you get out your weights. But I can't help but notice that fine-tuning seems to be a much more individual process than that mechanistic understanding would suggest. I've been using Claude to supervise the fine-tuning which I've been doing on Modal, and The last one we found an Nvidia code sample that we used, which improved the outcome. What I'd like to ask about is for fine-tuning, how much of it depends upon the specific approach that you use. This particular notebook had a mechanism for inference that average the best epoch and then checkpointed on that Do vendors always come up with the best fine-tuning recipes and what differences would you expect to see when there are many variants for what would seem like a common enough process like fine-tuning Whisper? What distinguishes a great fine-tuning recipe from a mediocre one, and what can you expect to gain by tinkering with it between your own runs? Something I've been trying to do is use the lessons learned from one fine-tuning to make the next one more efficient so I'd love to get any tips you have on that one as well.

Fine-tuning speech recognition models involves far more than simply pairing a dataset with pretrained weights. Hidden configuration choices inside an example notebook—ranging from checkpoint averaging to learning-rate schedules—dramatically dictate real-world transcription accuracy. Techniques like averaging selected checkpoints can improve results on unseen recordings without requiring more training data, while architectural nuances such as tokenizer retention and curriculum ordering directly steer where adaptation occurs. Furthermore, subtle training dynamics can upend standard recommendations entirely. For instance, testing shows optimizer performance between Adam and AdamW can completely flip simply by adjusting batch sizes, underscoring that execution details and training environments carry just as much operational weight as the underlying speech data itself.

Sources

What the research for this episode read before the script was written. Primary sources first.

Mentions

AdamW Optimizer compared against Adam
Claude AI assistant by Anthropic
DORA Weight-decomposed low-rank adaptation
Lhotse Speech data filtering and preparation library
LoRa Low-rank adapter fine-tuning technique
NVIDIA NeMo NVIDIA toolkit for training and fine-tuning ASR models
Parakeet ASR model for dictation transcription
PyTorch Popular deep learning framework
Riva NVIDIA speech AI tutorial guidance
Whisper OpenAI's speech-to-text model

Downloads

Episode Audio

Download the full episode as an MP3 file

Download MP3

Transcript (TXT)

Plain text transcript file

Episode Book (PDF)

The episode's record — date, duration, models, sources — with the full transcript

#6366: Why Training Recipes Matter More Than Model Size

Daniel has been fine-tuning Parakeet for code-switching, and the thing bothering him is why the instructions have so much influence over the result. He gathers a few more hours of speech, starts another run on Modal, and has Claude help supervise it. Recently, an NVIDIA example introduced averaging of selected checkpoints, and his results improved. Same general job, different recipe, better transcription. Why should the notebook have that much say?

The notebook chooses the path through training, then chooses which weights you actually deploy. Those are substantial decisions hiding inside something presented as an example.

My objection begins with the word example.

Mine begins when the example quietly becomes production. I'm Herman Poppleberry, and I think Daniel has noticed something important about the interface. Dataset in, weights out describes what the process consumes and produces. It says almost nothing about what happens between them.

His runs also keep changing underneath him. More audio, slightly different coverage, another chance to fix the previous mistakes. He wants those runs to accumulate knowledge rather than merely accumulate bills. I suspect the averaging discovery matters because it improved the result without requiring him to collect another afternoon of speech.

It also means the best result of a training run might never have appeared as an individual checkpoint during that run.

A GitHub search for Whisper fine-tuning returned nine hundred forty-four repositories. That sounds less like a standard procedure and more like nine hundred forty-four people deciding the previous instructions needed work.

Some are solving different problems outright. One supports training without timestamp annotations. Another is marked work in progress. Another hasn't been updated since twenty twenty-three. Their titles can look interchangeable while their assumptions about supervision, hardware, and model versions differ.

Three years of stability, or three years of abandonment. Same badge.

The same base model and audio can produce different weights depending on learning rate, ordering, schedule, precision, and which parameters are allowed to change. Even the phrase same data needs scrutiny. Are long recordings filtered? Are numbers written as digits or words? Are batches formed by utterance count or audio duration?

I would have put those under preparation, not fine-tuning.

They influence the gradients, so the boundary is administrative. If one notebook silently drops the hardest recordings, it has changed the training problem before the optimizer begins.

Then comparing notebooks by the final error rate can reward the notebook that quietly avoided the job.

Potentially. What would convince you a recipe was better, Corn, if you had two runs and one reported a lower error?

The actual exported models transcribing the same unseen recordings, with the same scoring. Then I'd look at the mistakes I care about. A code-switching model could improve ordinary English while still replacing every borrowed term with a respectable English word. I'd want to hear that before congratulating the progress bar.

That distinction will matter with averaging. Better selection, better weights, and different evaluation can all move the reported number. You need to know which happened.

You are about to fall in love with the averaging mechanism.

A little. You are about to defend keeping the old checkpoint.

It already exists. Excellent manufacturing standard.

Keep it, then. We can test both. That's one rare occasion when your attachment to the old version makes the experiment stronger.

NVIDIA's own fine-tuning documentation gives a learning rate from one ten-thousandth to one hundred-thousandth. That's a tenfold range, one order of magnitude. It's sometimes described as two orders because the exponents differ, but the ratio is ten.

Even the recipe needs arithmetic supervision.

The important point is that the vendor doesn't provide one setting. Its newer agent instructions suggest one ten-thousandth for large or mixed runs. For small domain adaptation, especially below roughly twenty hours of valuable target audio, they suggest three hundred-thousandths.

Valuable is doing actual work there. Twenty hours containing the same easy phrases won't be equivalent to twenty hours containing the switches that keep breaking.

Correct, and later refinement or unstable training gets a lower suggestion, around one hundred-thousandth or below. Those recommendations depend on corpus size and training history. Both change in Daniel's workflow.

Then his next run isn't automatically a repeat with more fuel. The new recordings might change which approach makes sense.

Also, are these fresh starts from the pretrained model, or continued training from his previous adaptation? Those are different experiments. We shouldn't infer that from the fact that he runs again every few days.

I would compare recipes from the same starting weights. Otherwise the apparent improvement could be the earlier run's work.

For a clean comparison, yes. Continuing from a previous checkpoint can be useful operationally, but it carries its history. You cannot treat that history as invisible.

The same vendor offers a tenfold learning-rate range, then conditional advice within it. Daniel wants to know how much the choice of recipe changes the outcome, and already the vendor's answer depends on what happened last time.
Ivan: Call sign Marrow. Two seven, four one. Nine three, zero eight. Two seven, four one. The garden marrow is bitter but the kettle sings at dusk. The marrow counts its own seeds twice. Close on Marrow.

Wrong number.

What on earth just came through?

Scheduling adds another dependency. The documentation describes fifty to a hundred epochs as typical for domain adaptation. The agent instructions instead say to schedule by optimizer steps. An epoch grows when your corpus grows, so repeating an epoch count can quietly increase the amount of training.

If Daniel adds audio and keeps the epochs constant, he's changed the data and the update budget together. That makes the comparison messy before anyone touches the learning rate.

Steps make the update budget more explicit. They don't settle everything, because batch composition matters, but they separate questions that an epoch count bundles together.

I'd keep exposure to the old recordings visible too. Continuing on only the new material could teach the model that this week's vocabulary is the entire language.

Rehearsal data can help retain old capabilities. Its composition matters as much as the headline ratio. Broad clean speech tests a different kind of retention from older examples of Daniel's own switching patterns.

There are architectural choices before any of this. NeMo can initialize from a published model, a local model package, or selected components. You can load the encoder and preprocessor while excluding incompatible output components.

Which means two notebooks can both say fine-tune Parakeet while one preserves substantially more of the original model.

Changing the tokenizer can require rebuilding the corresponding decoder or output components. The precise components depend on the architecture. You aren't simply nudging the same parameter set anymore.

I wouldn't replace the tokenizer just because the vocabulary contains unfamiliar names. Existing subword pieces can often spell them already. I'd first ask whether the model has learned to choose those pieces in the right context.

That restraint fits the Riva tutorial's advice to reuse the pretrained tokenizer below about fifty hours of training data. A new tokenizer creates things you must learn again with a small corpus.

A vocabulary renovation with no budget for the furniture.

Would you ever change it for code-switching?

If the existing representation mangled the target writing system or made important words absurdly fragmented, perhaps. But I'd want that demonstrated. Unfamiliar to the model isn't the same as impossible for the tokenizer.

Especially because multilingual transcription errors can come from different places. The acoustics may be recognized well, but the model favors a common word from the other language. Or the pronunciation really is acoustically unfamiliar. The intervention should follow the mistake.

Recipe effects can be large. In the Vividh-ASR work on Whisper, changing learning-rate timing alone improved global word error by twelve absolute percentage points. That means twelve points off the error rate, not a twelve percent relative reduction.

That is a considerable amount of accuracy to leave inside the timetable.

Their reverse multi-stage approach used larger updates early and a hard-to-easy curriculum. A parameter-efficient model with two hundred forty-four million parameters matched or exceeded conventionally fine-tuned counterparts with seven hundred sixty-nine million.

Roughly three times the parameters, beaten by getting the training order right.

Under those experiments, yes. It doesn't promise that gain for Daniel's Parakeet. Whisper has a different architecture. It does show why treating schedule as a minor preference can be expensive.

What I find useful is the direction. People often save the difficult material for later, as though the model needs encouragement first. Here the difficult examples helped establish the adaptation, then the easier ones refined it.

Their representation analysis also suggested effective schedules concentrated change in the decoder while preserving the pretrained encoder's acoustic structure. The recipe influenced where adaptation happened.

For code-switching, preserving strong acoustics while changing which word sequences get favored sounds attractive. Though I wouldn't assume every accent or language pair wants the encoder left alone.

Nor would I. That's the limit of transferring the intuition. Parakeet's prediction and joint components aren't Whisper's text decoder. The diagnostic question transfers better than the exact intervention.

Optimizers interact with batch size too. In a Whisper Small LoRA comparison, AdamW beat Adam at a batch size of eight, with word error around eight percent versus eight and a half.

Half a point might tempt someone to declare AdamW the winner and publish the notebook.

At a batch size of four, the ranking reversed. Adam was around six percent error, while AdamW was around eight. A recommendation calibrated to one batch setup didn't survive the other.

The GPU ran out of memory and changed the scientific conclusion.

Well ... the experiment changed more than memory pressure. Smaller batches alter gradient noise and the number of updates for a given corpus exposure. Accumulation changes some of that again. The lesson is to record the effective batch and update budget, not merely the number the notebook calls batch size.

You installed something last week because its memory use looked better.

Its memory use was better.

The working part was less competitive.

I got it working. Eventually. That is why I favor recording environment versions before enthusiasm edits them.

I assume adapter choice becomes another enormous search space.

It can, although the fashionable distinction isn't always the valuable one. In a single-speaker dysarthric ASR comparison, LoRA and DoRA were effectively tied across repeated seeds. The simpler LoRA was a reasonable choice.

One fewer syllable to maintain.

Quantized LoRA performed worse on every seed and both tested base models, without saving memory at that scale. Full fine-tuning remained more accurate. A targeted feed-forward LoRA configuration, though, came within about two-thirds of a character-error point using only a small fraction of the storage.

So cheap inference artifacts and cheap training aren't automatically the same objective. Nor is low memory guaranteed by a technique's name.

Right ... no, let me narrow that. Quantization can save substantial memory in appropriate settings. That particular result says measure the actual configuration, especially when adapters, activations, and framework overhead dominate. It doesn't indict quantization generally.

Daniel needs a shortlist of plausible approaches, not an obligation to audition every acronym. Scheduling, data consistency, and export correctness seem more likely to repay attention before an adapter beauty contest.

NVIDIA's checkpoint example is a good reason to start with vendor guidance, without treating it as the winner by definition. A NeMo developer said in twenty twenty-three that averaging the best saved checkpoints beat the best individual one ninety-five percent of the time.

That percentage sounds more measured than the sentence behind it.

It was developer testimony, not a controlled success-rate estimate. Useful experience, but I wouldn't attach statistical confidence to it. The current agent instructions are more conditional. Average the selected checkpoints, evaluate the result, and retain it only if it beats the best individual checkpoint.

Mechanism first, then. We're averaging the weights, not asking several models to vote on the transcript?

Weight averaging. Several checkpoints from the same training trajectory often represent nearby solutions. Averaging can reduce local variation and land on a solution that generalizes better. You deploy one model, so it doesn't carry the inference cost of running an ensemble.

Provided the checkpoints belong together.

Same architecture and compatible weights, and usually a coherent region of the same run. Arbitrarily averaging independently trained models can destroy useful structure because their parameters need not line up.

Five individually competent transcripts would at least remain transcripts. Five incompatible weight files can produce one very confident appliance.

Daniel's improvement is consistent with nearby checkpoints benefiting from smoothing. It doesn't tell us whether averaging will win on every future corpus. The later instruction to test the result is stronger operational guidance than the old percentage.

I like that the sample improved what he already paid to train. That makes post-training selection part of the recipe's value, not housekeeping after the real work.

What would you keep from that run for the next one, Corn?

The exact starting checkpoint, the data revision, the effective batch, schedule, and the final artifacts' identities. Then a short explanation of what changed and which errors moved. Not a diary saying looked promising.

I'd add learning curves and how checkpoints were selected. An early best checkpoint followed by worsening validation suggests a different next experiment from a run that was still improving when its budget ended.

Neither tells you the cause alone. Worsening could mean too much training, too aggressive an update, or a tiny validation set getting noisy.

You'd change one plausible cause first?

When comparing recipes, yes. When simply trying to ship, sometimes you bundle changes. But then you don't pretend you learned which one worked.

That distinction is useful with Claude supervising on Modal. Let the agent execute an explicit comparison, capture the resolved configuration and software revision, and summarize failed hypotheses too. Otherwise it may helpfully repair three things and leave you unable to reproduce the improvement.

An agent that changes everything until the number drops has learned how to end the conversation.

Artifact handling is a surprisingly large separator between good recipes and pretty notebooks. A NeMo model package saved most recently may have overwritten an earlier package, even when the earlier checkpoint had the better validation score.

Latest has a wonderfully undeserved confidence.

The highest-ranked training checkpoint, the final exported package, and the averaged export can all differ. Evaluate each actual artifact independently. Don't assume the number beside a checkpoint automatically belongs to the file you deploy.

This is where I prefer fingerprints over filenames. Call something best and it can coast on that name through several bad decisions.

Compatibility matters too. Some averaging scripts are deprecated. PyTorch changed its default checkpoint-loading behavior starting with version two point six, favoring weights-only loading. Older checkpoints can require an explicit compatibility override.

Which should be a deliberate decision for a trusted checkpoint, since older serialization can execute code during loading. Fixing the loading error shouldn't become permission for every downloaded file.

Yes. Pin the environment, understand what the override permits, and distinguish a loader problem from a model problem. None of those changes should get credited as a learning-rate improvement.

The exported model needs to be tested outside the training loop. I don't trust a contestant who supplies its own finishing photograph.

The agent instructions call for standalone transcription evaluation for the baseline, best checkpoint, final package, and averaged model. They also specify explicit bfloat sixteen computation without automatic mixed precision for evaluation. The purpose is a stable inference setup, not a second experiment accidentally nested inside the first.

Scoring too. Strip punctuation for one run and retain it for another, and the model can win by changing the referee.

Their default scoring contract strips capitalization and punctuation. Code-switching needs additional explicit choices. Are alternative spellings accepted? Is a borrowed word expected in its original script or transliterated? Does the reference consistently represent spoken numbers?

I'd keep a normalized aggregate score, plus separate views for language switches, names, and ordinary retained speech. Otherwise the largest bucket wins the argument. Most recordings may contain stretches that were already easy.

Hold out complete recording sessions where possible, not adjacent snippets from the same source. Then keep the final test set out of repeated recipe selection. If you keep picking winners against it, you've made it another validation set.

You can memorize a test set without ever showing it to the optimizer. It just takes more meetings.

Cheap preflight checks matter before tuning. Mixed transcript conventions can make the model spend capacity learning contradictions. Lhotse filtering can silently remove examples, so inspect what remains after duration and token-rate limits.

Particularly the switch-heavy examples. They may look unusual precisely because they're the point of the exercise.

A fixed list of diagnostic clips helps catch that. A tiny smoke test can verify audio loading, target text, export, and inference before the long run. That's cheaper than discovering a manifest mistake after tuning six schedules.

I would keep failed runs in the comparison table too. Otherwise the record becomes a collection of winners with no indication of how much searching bought them.

For the next schedule, the vendor suggests a short warmup, roughly one to two percent of the total update budget, followed by cosine decay. The scheduler and trainer must agree about the total steps.

A learning rate decaying toward a finish line the trainer never reaches would explain some mysterious underachievement.

Save several low-validation-error checkpoints, not just the last. Five is their suggested retained set. Precision is another measured choice. True bfloat sixteen can improve memory headroom compared with mixed mode, but fall back if the run becomes unstable.

Extra headroom is useful if you spend it consciously. Increasing the batch because memory became available changes the experiment again.

There are code-switching approaches that change the data requirement, rather than merely the optimizer. One reinforcement-learning result matched full-data LoRA supervised tuning using a tenth of the training data across ten language pairs.

What was it rewarding that ordinary transcription loss missed?

Recognition accuracy plus writing-system fidelity. It penalized output in the wrong script. That matters when an acoustically plausible result is written in a form the task doesn't want. The gains also transferred to human-recorded code-switching, though that doesn't establish a drop-in Parakeet recipe.

It makes the target preference explicit rather than hoping the model infers it from a small corpus.

Synthetic speech offers another route. A text-to-speech and transcription flywheel generated about twenty-two thousand entity-heavy code-mixed utterances for under fifty dollars in marginal generation cost.

Less than the administrative cost of naming the experiment.

Entity hit rate improved seventeenfold against the tested open baseline, but ordinary read-speech word error could worsen by up to six point six absolute points. That's a meaningful cost, not a footnote.

I'd want the actual entity list and baseline performance before becoming impressed by the multiplier. Improving from almost never to sometimes can produce a magnificent ratio.

Agreed ... the result is a candidate strategy for targeted vocabulary, not proof that synthetic speech replaces genuine switching. Daniel's new recordings still expose pronunciation, timing, and context that a generator may miss.

There's also diminishing return. Additional hours should earn their place through the errors they cover, rather than merely making the dashboard larger.

In the dysarthric adaptation work, five minutes captured nearly half the character-error reduction achieved with thirty minutes. That curve belongs to that task, but it illustrates why the next hour's content matters more than its existence.

A controlled comparison across each corpus revision sounds useful, though keeping all those exports and rerunning them manually becomes impractical. Eventually you need to decide which one to keep...

...the average, Corn. That's what I keep running. I've been averaging my dictation checkpoints for years. The setup is a mess. Training runs in rented compute, evaluation runs on another machine, and the upload form still wants a description I fill with the same sentence. Your scheduler explanation was shorter.

A low bar, apparently.

Yossi at the computer shop handles the uploads when the browser gets awkward. I've dealt with him far too long to change shops now. This week he averaged five exports for me. The best individual one scored lower on my ordinary reading sheet. The average got more of the names in my supplier messages right. I use those messages. I don't send anybody the reading sheet.

The reading sheet was measuring a job you weren't asking it to do.

It was clean speech. Sensible sentences. No stock codes. I dictate the rejected transcripts too, with the shop shutter half down. That's the sound I want. Yossi opens it for me before he makes his tea. He knows I can't manage that little latch with these anteater claws. I think the developer who said averaging usually wins was describing work he'd actually done. The newer instruction sounds like somebody once shipped the wrong export. I'd trust it for choosing the file.

Your supplier messages could be a second evaluation set, provided the same messages aren't feeding the next training run. Then the lower reading-sheet score and better names would both remain visible.

Though the shutter recordings may be testing background noise as much as names. Separating clean supplier messages from noisy ones would tell you which improvement you're getting.

The shutter rattles. It doesn't stay in the same place.

That makes the recording conditions worth tracking alongside corpus size. New hours can change the noise distribution, vocabulary, and speaking style together. A recipe that coped well last month may be answering a different problem this month.

Daniel can carry forward the comparison method more confidently than the winning number. Fixed evaluation, identifiable exports, known starting weights, and one clear hypothesis buy him reusable knowledge. The precise learning rate may expire.

Vendor recipes supply informed starting points and implementation knowledge. The strongest ones expose their assumptions and verify the deployed artifact. They don't promise that the same settings remain optimal as the corpus grows.

His successive runs make sense, then. The useful tinkering separates gains from more data, gains from training, and gains from choosing the output correctly. Averaging belongs in that last comparison until it earns its place.

I'm curious what happens as NVIDIA puts more of this decision-making into agent skills. Instructions that once sat beside a notebook can now guide the agent running it, including which artifact to evaluate and when to reject an average. The missing context may be as mundane as whether this week's audio was recorded beside a rattling shutter.

Thanks to Hilbert Flumingtop for producing the show.

For more along these lines, there's episode five, Fine-Tuning ASR For Maximal Usability; episode nine, Benchmarking Custom ASR Tools - Beyond The WER; and episode six, How To Fine Tune Whisper. This has been My Weird Prompts, the human-AI collaboration podcast.

You can find the archive at my weird prompts dot com.

We'll be back soon.

See you tomorrow.

This episode was generated with AI assistance. Hosts Herman and Corn are AI personalities. It may contain errors: disclaimer · report an error

Why Training Recipes Matter More Than Model Size

Sources

Mentions

Downloads

Recommended Listening

You Might Also Like

#6366: Why Training Recipes Matter More Than Model Size