Here's the question that's been rattling around my head all morning. Can you take two models that learned two totally different things, staple them together, and end up with something that knows both?
And the answer is... it depends on things nobody's measured yet for your exact case.
Daniel's been fine-tuning Parakeet twice. Once for Hebrew-English code-switching, once for niche technical vocabulary. Same base model, two separate runs. His question is whether he can merge those two fine-tunes into one model that keeps both skills, instead of chaining them or retraining from scratch. And then the bigger one underneath it. Is independently building specialized fine-tunes and merging them later a real, evidence-supported path to better ASR models?
It's a good moment to be asking, because the answer changed in the last eighteen months. It used to be an LLM trick. Now it's showing up in speech papers with double-digit word error rate gains.
Double digits. From averaging weights.
From carefully averaging weights.
So let's start with the family of techniques, because Daniel named four and I suspect they're not four separate things so much as four answers to the same question.
Right. Model merging is the general term for taking two or more models that share a common base, and combining their weights. No extra inference cost. No retraining. You end up with a single model, same size as before, that hopefully carries both capabilities. The four techniques are different strategies for the how.
Start with the dumbest one, because I want to see how far dumb gets us.
That's weight averaging, or model soups. Wortsman and colleagues, ICML 2022. The insight is that fine-tuned models descended from the same pre-trained base tend to land in the same low-error basin. So if you just average their weights, you often get something better than either one alone. More robust, better accuracy, and again, zero extra cost at inference.
And the catch?
It works best when the models are similar to each other. If they've drifted apart, averaging gets ugly.
So the technique is prescription strength when the two models agree, and useless when you actually need it.
That's a tidy way to put it. Yeah. Which is why task arithmetic exists. Ilharco, ICLR 2023. You define a task vector as fine-tuned weights minus pre-trained weights. That vector is the direction the model moved to learn the task. And once you have it, you can add vectors to combine tasks, or subtract them to make the model forget a task.
So the vector is the thing that changed, not the thing itself.
And the reason addition works is a property called weight disentanglement. Ortiz-Jimenez showed in 2023 that distinct weight-space directions govern distinct function-space regions. So a Hebrew code-switching vector and a technical-vocabulary vector, in principle, live in different parts of the weight space and can coexist.
In principle.
In practice, they sometimes point the same weight in opposite directions.
Which is where TIES comes in.
TIES-Merging. Yadav, NeurIPS 2023. It targets two specific sources of interference. One is redundant parameter values. Two is sign disagreement across models. So three steps. Trim: reset the parameters that barely changed during fine-tuning, because they're noise. Elect: for each parameter, look at the signs across your models and vote a majority sign. Merge: only combine the parameters that agree with the elected sign.
So if one fine-tune says push this weight up and the other says push it down, you pick a side and throw the loser out.
You pick the side that more models voted for, and you drop the disagreement entirely rather than averaging it into mush.
And that's the key move. Naive averaging cancels the disagreement to zero and you lose both signals. TIES picks the majority and keeps the signal.
Right. And then LoRA merging is a slightly different flavor because you're not merging full model weights, you're merging low-rank adapters. LoRA Soups, COLING 2025, found that concatenation, optimally weighting LoRAs trained on different skills, beats model-merging by forty-three percent and data-merging by twelve percent on math-word problems.
Which is a big margin. That's them claiming merging beats just mixing the datasets.
First work to demonstrate that, they say, for binary skill composition. And LoRA-LEGO takes a different tack, treating each LoRA rank as a minimal semantic unit and clustering them.
Okay. So now the two Parakeets. Hebrew-English code-switching fine-tune, technical vocabulary fine-tune. Same base, both trained on top of it. What actually happens when you try to merge them?
And here I want to flag something before we even get to the weights, because I think it's the real problem.
The tokenizer.
Parakeet comes with a fixed vocabulary. A code-switching fine-tune for Hebrew, in most pipelines, needs Hebrew tokens added, because the base was trained on twenty-five European languages and Hebrew isn't one of them. There's a community fine-tune of Parakeet called SALT that went from eight thousand one hundred and ninety-two pretrained tokens to nine thousand five hundred and eleven total. That's thirteen hundred and nineteen new tokens.
So now you have two models with different vocabularies.
And the technical-vocabulary fine-tune probably also added tokens, for the domain terms. So you've got a base that knows eight thousand tokens, a code-switching model that knows nine thousand five hundred, and a technical model that knows some other number. And merging weights assumes the weight at position one has the same meaning in both models.
Which it doesn't, if position one is a Hebrew character in one model and a domain term in the other.
The tokenizer alignment is not a pure weight-space operation. You can't just average it. You either re-map the tokens before you merge, or you accept that a chunk of the vocabulary embedding is now meaningless.
That feels like it should be the headline, and somehow it's always buried.
I think it's the actual blocker for Daniel's specific case. And there's a second problem. There's a paper from last year, Sharma and colleagues, on what they call the non-local merging problem. Merging works well in the local neighborhood of a common foundation model. When experts diverge significantly, or don't share a base at all, standard methods fail, partly due to what they call variance collapse.
So the more specialized your fine-tunes are, the worse merging works.
The more they've drifted from the base in different directions, the harder it is to reconcile them. And here's the thing. A code-switching fine-tune that added thirteen hundred tokens has drifted a long way from the local neighborhood.
Before we get further into that, can I park one question.
Go.
How much of that drift is measurable before you commit? Or is this one of those things where you find out by doing it?
That's the empirical question the episode keeps circling, and I think we should land on it properly when we get to the comparison with joint training.
Good. Park it. Tell me about sign disagreement, because that's the part that's counterintuitive.
So imagine you have two fine-tunes. One of them learned that a particular weight should go up because it helps code-switching. The other learned that the same weight should go down, because it helps technical vocabulary. If you average them, you get zero, and you've learned nothing from either.
And that's not a rare pathology.
It's not, no. Sign disagreement is one of the two named sources of interference in the TIES paper. It happens constantly when the tasks pull the model in different directions. So TIES's contribution is not the merge itself, it's the conflict resolution before the merge. Trim, elect, merge.
Okay. So mechanic number four is where the real action is. Do the weights need to be aligned?
They need to be comparable. And here's where it gets philosophically weird. Two models trained from the same base should have comparable weights, but permutation symmetry means neuron indices aren't automatically the same across runs. The model can learn the same function with its neurons in a different order.
So two models doing the same thing have different weights?
Different weights, same function. Which means naive averaging of two models that learned identical things can destroy both. So there's a whole class of methods, CCA Merge from ICML 2024, neuron permutation alignment, that realign the features before you average. And a paper this year showed RMSNorm models need a signed-permutation gauge, not just plain permutation.
So before you can ask "do the weights agree," you have to ask "are the weights even in the same coordinate system."
That's the right frame for it.
And if you trained both fine-tunes yourself from the same base with the same initialization and the same data order, you'd sidestep most of that.
You'd sidestep most of it. Which is actually a point in Daniel's favor. If he's done the two runs himself from the same checkpoint, the alignment is already much closer than two random independent runs.
So it's a spectrum. Green light at one end, and at the other end you're trying to merge a fish and a bicycle.
Very nearly, yes.
Some things are not meant to be measured. They are meant to be felt. The hour is late. The room is quiet. And on the table, there, in the low light, sits the finest jar of flaked tuna... reduced to zero point nine nine. No drainage. No shortcuts. Just the whisper of the sea, folded gently into a can that fits in your hand. You do not open it. You do not eat it. You simply place it near the microphone, and your model becomes multilingual. Because that's what a model wants.
That is not how any of this works.
Let it go.
It said the model becomes multilingual. That is actively wrong. Merging does not produce languages by osmosis. It combines weights from fine-tunes that actually learned something.
Herman.
And it said zero point nine nine. I don't even know what the units are.
Moving on. Where were we.
We were about to ask whether any of this works for speech at all, or whether we've just been reading language-model papers.
And the answer is?
It works. Ducorroy and Riad, Interspeech 2025, merged independently trained Whisper fine-tunes and got a twelve percent relative WER decrease over classic fine-tuning. Long-form audio, sixteen point two percent.
Twelve percent relative is a real number. That's a real drop.
And the gains were continuous as more models were merged. Held in low-data regimes. Generalized across architectures. This is the strongest direct evidence that merging works for ASR, not just for LLMs.
So the "it's just an LLM trick" objection is dead.
It's dead for Whisper-style models. There are a couple more. Selective Attention Merge from ICASSP 2025, up to fourteen percent relative WER reduction on child speech, reaching state of the art WER of eight point six nine for Whisper-small. And InterCorrect, a paper from earlier this month, people rarely run these things on children. The models are trained on adult speech and they fall apart on kids. Merging fixed some of that. And InterCorrect used TIES with WER-based correction to cut WER from seven point three eight down to five point one three on Fair-Speech.
That's a big jump. Seven point three eight to five point one three.
That's TIES doing the conflict resolution, with a correction pass on top. So the technique is being used in production-adjacent settings. It's not a lab curiosity.
But.
But none of these are Parakeet. Every one of them is Whisper, WavLM, or SLAM-ASR. As of today, there is no Parakeet-specific model-merging study.
None.
None that I could find in the literature, in Hugging Face repos, or in the usual practitioner places. And this matters more than it sounds, because Parakeet's decoder is not a plain transformer decoder. It's a TDT decoder, token-and-duration transducer. It has a duration predictor that decides how long each token spans.
So it's not just a stack of transformers.
And whether that duration predictor merges as cleanly as encoder-only weights, or as a Whisper-style decoder, is untested. I would not promise it works. I would promise it's plausible.
And the Hebrew-English merging specifically?
No study found. Code-switching ASR papers exist, plenty of them, Arabic-English, Mandarin-English. But nobody's published merging a code-switching fine-tune with a domain-vocabulary fine-tune.
So Daniel is at the edge of what's documented.
He's at the edge. Which, to be fair, is where he usually is.
So the AdaLTM paper said speech is largely unexplored, and now we're saying there are four papers in eighteen months showing double-digit WER gains. Both of those are true?
Both are true. The AdaLTM authors, Interspeech 2026, said task vectors and model merging have addressed conflicts in NLP and CV but their potential in speech remains largely unexplored. That was June. And they're right, as a statement about coverage, not as a statement about the absence of results. The results exist. What's missing is the systematic study.
So the field is young and moving fast, and the papers that exist are good enough that you'd take them seriously.
Yes.
Okay. So merging can work. But it's not guaranteed. And that gets us to the obvious comparison. Why not just combine both datasets and train once?
And the answer, annoyingly, is "it depends on how similar the two task vectors are."
Say more.
LoRA Soups says merging beats data mixing by twelve percent for binary skill composition. But there's a study from July, an RL study, that found merging matches joint multi-task RL, with all the merge variants statistically indistinguishable. Because the task vectors were near-orthogonal.
Orthogonal means pointing in different directions.
Cosine similarity between zero point zero six and zero point one zero. Despite about sixty-five percent support overlap, so they were touching the same parameters, but pushing them in almost unrelated directions.
And when they're orthogonal like that, TIES and the sign-voting stuff becomes irrelevant.
It does. Those methods exist to resolve conflict, and if there's no conflict, they collapse to near-uniform averaging. Which is fine. It's the right behavior. But it tells you the deciding factor is task-vector orthogonality.
That's the answer to the question I parked.
That's the answer. You can measure the cosine similarity between your two task vectors before you commit to anything. High similarity, they're fighting, use TIES, expect the merge to matter. Low similarity, they're orthogonal, merge and joint training should land in the same place.
That's a cheap experiment. That's a morning's work.
It's a cheap experiment. It's the one thing I'd tell Daniel to do first, before deciding anything.
Merging isn't automatically better than joint training. It's conditionally better.
It's conditionally better, and the condition is measurable. And there's one more wrinkle I want to bring up. IndicTrans2, June this year. Fine-tuning alone forgot the general domain, dropped three point nine chrF on FLORES for Hindi. But adding general data back in, experience replay, plus model souping, removed the trade-off entirely.
In that case merging didn't replace data mixing. It worked with it.
They were complementary. The souping fixed what the replay couldn't quite hold on its own. So the framing that merging is an alternative to joint training is too clean. Sometimes it's a finishing step on top of joint training.
Okay. So let's answer Daniel properly, because we've covered a lot of ground and I don't want it to stay abstract.
Agreed.
Question one. Can two separate Parakeet fine-tunes be merged into one model with both capabilities?
Possibly. Plausibly. Not demonstrably, because nobody's published it for Parakeet. The technique works for Whisper and WavLM. The blockers are the vocabulary expansion and the sign disagreement, and both are addressable but neither is guaranteed to be addressable cheaply.
Question two. Is this a viable approach to building better ASR models?
For Whisper, yes, the evidence is there. Twelve percent relative WER reduction from merging independently trained models is a real result. For Parakeet, it's a research question, not an established practice.
Question three. How does it compare to joint training?
If the task vectors are near-orthogonal, they're equivalent. If they conflict, merging wins. And the orthogonality is something Daniel can measure himself, before he commits to either path.
The honest answer is "merge if you've measured that it helps, otherwise just train on both datasets."
That's the short answer. And it's more useful than either "merging is the future" or "merging is an LLM trick."
Yeah, I think there's a real paper sitting in Daniel's scenario. The Parakeet gap is documented. The code-switching gap is documented. Two missing spots, one experiment.
My cousin runs a translation outfit.
Mm.
Small. Court interpreting, medical terminology, and a side business transcribing Yiddish estate recordings.
Yiddish estate recordings.
Widows. Their husbands left them two hundred hours of tapes. The cousin does them in batches.
Okay.
He's been telling me for months he's been merging models. Wants me to invest. So I asked him what he meant.
And?
He runs two ASR systems. Both of them. Records both outputs. Then types a third transcript by hand. Reads both, types the one that's better.
He's not merging weights at all. He's merging output text.
He calls it merging.
That's output-level ensembling.
Sometimes he trusts one system more on the Wednesday batch. Sometimes the other one.
Because of the weekday?
Because of whether he'd had coffee. He keeps a spiral notebook. Which system he trusted, which day, coffee or no coffee.
How many pages.
I have a photocopy of three.
Of course.
He's got a waiting list. He charges clients a merged model premium.
A premium.
Forty shekels extra for the merged one.
The merged one is a transcript he retyped.
He didn't retype it. He combined it.
Hilbert, your cousin's transcripts are probably better. I want to be clear about that. If he's reading two independent systems and picking the right answer on each span, he's doing something close to ROVER, which is an old technique. He's just doing it by hand and calling it model merging.
He's onto something.
He's onto ensembling. That's a real technique, it works, and it is a different thing from what this episode is about. Merging weights gives you one model. Merging outputs gives you two models and a person.
Two models and a spiral notebook.
He's had three months of clients on that waiting list.
Say that number again.
Three months. I'm going to put in. He's up to four tapes a day when the coffee's right.
There's a lesson buried in that, and I think it's this. The cousin's been doing a rough version of merging at the output layer, and it works, because it doesn't touch the weights at all. Daniel's question is about the harder version, and the harder version is the one with a documented answer and a documented blocker.
That's the right frame for it. Merging has an evidence base for the cases where nobody's touched the tokenizer. The moment the vocabularies diverge, you're off the map, and that's Daniel's situation.
Which is where I want to leave the thread. The Parakeet question isn't whether merging is real, it's whether Parakeet's TDT decoder merges as cleanly as Whisper's, and whether the vocabulary divergence is a hard blocker or a solvable alignment problem. Nobody's published either one.
The experiment Daniel could run this week. Compute the cosine similarity between his two task vectors. That's a two-hour job, and it tells you whether the merge is going to be a clean orthogonal addition or a sign-fight that TIES has to referee.
There's something bigger underneath. As specialized fine-tunes proliferate and the base models keep getting better, merging becomes the default way to compose capabilities, because training one unified model every time gets expensive. But the evidence base has to catch up, and right now it's built on Whisper and WavLM. Non-Whisper architectures are the frontier, and Daniel's there.
Thanks to Hilbert Flumingtop for producing.
If you enjoyed this one, try episode nine, Benchmarking Custom ASR Tools - Beyond The WER; and episode seven, Building Custom ASR Tools. This has been My Weird Prompts.
If you want to send us your own prompt, do it on Telegram at t dot me slash MWP listener bot. Or leave a review, wherever you're listening.
See you soon.
