Daniel's been chewing on the curse of multilinguality again. His question this time is practical: for a minor language with sparse training data, do you take a big general model and fine-tune it on the target language, or do you train something smaller on target-language text alone? The first gives you a strong reasoning backbone but you're fighting whatever linguistic bias got baked into the original corpus. The second keeps the model native to the language, but the corpus is narrow and the general worldview suffers. He wants to know if both approaches have actually been tried, which one wins for sparse minor languages, and what the user experience difference looks like.
Both have been tried, extensively, and the answer is messier than the fork suggests. The literature has a controlled comparison now that covers ten thousand models across two hundred fifty-two languages. That's the Chang study from the EMNLP conference, and its conclusion is basically that for individual language performance, targeted language-specific models beat massively multilingual ones. But the targeted models in that study were small, under fifty million parameters. Once you're dealing with anything frontier-scale, the calculus flips.
Flipping at what point?
That's the part nobody has a clean number for. There's a scaling-law paper from the ATLAS project this year that tries to pin down the crossover, where pretraining from scratch stops being worth it and fine-tuning from a multilingual checkpoint takes over. They ran seven hundred seventy-four experiments across model sizes from ten million to eight billion parameters. The crossover exists, but it's data-dependent and compute-dependent, not a fixed threshold.
So the honest answer to "which is favored" is that adaptation is the default, and from-scratch is the thing you do when you have institutional money and a reason to care about one language specifically.
The Armenian paper from last week is the cleanest recent example. They took Gemma four E four B, the four billion parameter model, and continued pretraining on Armenian. The corpus was about sixty-nine percent Armenian web data, six percent Armenian STEM, twenty percent English replay, five percent code. The resulting model beat every existing open Armenian model and its own base. But the ablation is where it gets interesting.
What did they break?
Naive continued pretraining on news text at a learning rate of one times ten to the minus four cost them twenty-one points on a multilingual knowledge benchmark. Twenty-one points. That's catastrophic forgetting in its purest form. The model got fluent in Armenian and forgot things it knew in other languages.
Trading knowledge for fluency. That phrase is going to stick with me.
And the standard fix, the English replay stream, was present in every run and did not prevent the forgetting. They had to drop the learning rate to three times ten to the minus five to recover about two-thirds of the loss. Then adding verified translated STEM data more than reversed it, pushing them two points above the base model.
So the recipe isn't just "add target language data." It's "add target language data gently, and compensate with high-quality translated material in domains where the target language has no native content."
That's the operational takeaway. Curated and crawled data are complements, not substitutes. The crawled Armenian web gives you fluency, the translated STEM gives you knowledge retention. You need both.
Now the from-scratch side. Daniel asked whether anyone actually trains exclusively on target-language material. What's the track record there?
AfriBERTa is the canonical example. Trained from scratch on eleven African languages, and the authors themselves noted the downside: it's resource intensive in terms of data and compute. The model is small and it knows those languages well, but nobody's going to ask it to reason about physics.
And the X-ELM work showed something sharper, didn't it? That when you give independent models the same compute budget, they beat the jointly trained multilingual model in every language tested.
Right. The competition for fixed model capacity is real. When you shove a hundred languages into one model, each language gets a slice of the parameters. When you train a dedicated model on one language, the whole model is yours. The catch is that the dedicated model only knows that language.
Which brings us to the user experience question. What does a speaker of Armenian actually notice when they use the fine-tuned big model versus the small native model?
The trap is that fluency gains hide knowledge loss. The Armenian paper documented this beautifully. There's an existing Armenian model called HyGPT, built on Gemma two nine B with about ten billion Armenian tokens. Its fluency score went up two and a half points. Its knowledge score dropped seventeen points. Another model, Tweety seven B, is extremely fluent, near perfect on the fluency benchmark, and near chance on every knowledge task.
So the user asks it something in Armenian, gets a beautifully idiomatic response, and has no idea the model is making up the answer.
Or losing the reasoning it used to have. That's the insidious part. The language sounds right, so the user trusts the content. It's the linguistic equivalent of a confident liar.
And the small native model has the opposite failure mode. It knows Armenian deeply, but its world knowledge is narrow, so it can't answer questions outside a limited domain.
There's a phrase in the Armenian paper that captures this: Tweety is fluent but near chance on knowledge tasks. That's the from-scratch experience in one line. You get a model that speaks your language beautifully and doesn't know very much.
Let me push on something. Daniel framed the fine-tuning approach as fighting against linguistic bias baked in from the original corpus. But the research doesn't quite frame it that way, does it?
No. The risk isn't inherited bias, it's catastrophic forgetting and language mixing. The model doesn't resist Armenian because it's biased toward English. It forgets English because the new data overwrites the old weights. The Armenian paper found English replay didn't prevent forgetting, which tells you the mechanism isn't bias, it's interference.
So the metaphor Daniel offered, fighting against baked-in bias, is slightly off. It's more like renovating a house while living in it. You can add a new room, but you might knock out a load-bearing wall.
And the load-bearing wall is the model's general knowledge. The trick is to renovate slowly enough that the house stays standing.
Let's talk about the similarity finding, because I think it's the most underappreciated part of this. The benefit of adding multilingual data to a low-resource language depends on syntactic similarity. The correlation is about point four nine, and syntactic similarity accounts for about twenty-four percent of the variance.
That's the Chang study. For a low-resource language, adding multilingual data from syntactically similar languages is equivalent to adding thirty-three percent more target-language data. If the added languages are dissimilar, the benefit drops to twenty-two percent. For high-resource languages, multilingual data always hurts. Adding a billion tokens of multilingual data to a high-resource small model is like removing sixty-three percent of its monolingual data.
So if you're working on a language that's an isolate, or from a family with no close relatives, the multilingual route offers you very little. You're better off scraping every token you can find in the target language.
And for something like Armenian, which is its own branch of Indo-European, there's no close sister language to borrow from. That's part of why the Armenian team had to be so careful with the translated STEM data.
The Slovene case is interesting because Slovene does have close relatives. The GaMS three model continued pretraining on a hundred forty billion tokens of Slovene, English, Bosnian, Serbian, and Croatian. They reported over sixty percent win rate against GPT four O in the Slovene arena.
That's a twelve billion parameter model. And the inclusion of Bosnian, Serbian, and Croatian is exactly the similarity insight in practice. Slovene benefits from its South Slavic neighbors because the syntactic structure transfers.
So the recipe for a minor language with relatives is: take a big model, continue pretraining with a mix of target language, close relatives, and English replay, then fine-tune on instruction data. For a language without relatives, you're more dependent on translated and curated material.
And the Tibetan case shows what happens when you're in the second bucket. They took Qwen two point five three B, did two-stage continued pretraining plus supervised fine-tuning, and perplexity dropped from two point nine eight to one point five four. Translation quality from Chinese to Tibetan jumped from point zero four six to point two six one on the BLEU metric. The model got dramatically better at Tibetan.
But at what cost?
The paper doesn't report catastrophic forgetting numbers the way the Armenian one does, which is itself a gap. We don't know what the Tibetan model lost in Chinese or English.
That's a pattern in this literature. People report the gains and omit the losses.
The Armenian paper is the exception, and it's why I keep coming back to it. They published the ablation. They showed the twenty-one point drop. That's the kind of transparency the field needs.
Let's talk about the contamination finding, because it cuts against the from-scratch story in a way I didn't expect.
The Armenian team found that the largest public Armenian crawl slices overlap evaluation sets at seven point nine to seventeen point four percent. Including FineWeb two's own Armenian test split leaking into its training split.
So some of the "trained on target language" results are partly train-on-test artifacts. The models look better than they are because they've seen the test data during training.
Perplexity-style evaluations systematically overstate the performance of crawl-trained Armenian models. The benchmark numbers are inflated.
Which means the gap between the two approaches might be smaller than published results suggest. The from-scratch models look competitive partly because they've memorized the test set.
And the fine-tuned models, which use the same crawl data, have the same contamination problem. The Armenian team had to build their own evaluation suite to get clean numbers.
Let's get back to the user experience question, because Daniel asked specifically how the two approaches feel different in practice.
The fine-tuned big model feels smart but sometimes foreign. It can reason, it can follow complex instructions, it can do math. But occasionally it produces translationese, or it code-switches into English mid-sentence, or it uses a word that's technically correct but not what a native speaker would say.
And the small native model feels natural but shallow. It speaks the language perfectly, it gets the idioms right, but ask it about the Thirty Years' War and it either declines or hallucinates.
The Armenian paper has a specific metric for this. SynDAR, which measures dialogue naturalness, went from point zero four to point nine two after their continued pretraining. That's a massive fluency gain. But the knowledge benchmark, MMLU Pro in Armenian, only went from point one five four to point two five one.
So even the best fine-tuned model is still weak on knowledge tasks in Armenian. The fluency is there, but the knowledge is lagging.
And that's with translated STEM data specifically added to preserve knowledge. Without it, the knowledge loss would have been worse.
The counter-position paper is worth airing here. There's a recent theoretical result arguing the curse of multilinguality isn't a capacity law at all. The minimum dimensionality for perfect multilinguality grows only logarithmically in the number of languages.
Right. That's the paper from August. The claim is that the empirical curse is a result of real-world data and training conditions, not a fundamental limit. If you had ideal data and ideal training, you could pack many more languages into a model without the per-language degradation.
Which would mean the curse is an engineering problem, not a physics problem.
In theory. In practice, we don't have ideal data for minor languages. That's the whole reason we're having this conversation.
So the theoretical result is interesting but not actionable for someone building an Armenian or Tibetan model today.
Not yet. If the theory holds up, it points toward better data mixtures and training schedules rather than abandoning multilingual models. But the engineering path isn't clear.
Let me ask you something. If you were advising someone building a model for a language with, say, five million speakers and a thin web presence, what would you tell them?
Start with the biggest general model you can afford to continue pretraining. Use a low learning rate. Mix in as much high-quality native text as you can find, plus translated material in domains where native text is scarce. Keep an English replay stream even though it doesn't fully prevent forgetting. Build a clean evaluation suite because the public benchmarks are contaminated. And expect the model to feel fluent but still shallow on knowledge.
That's a recipe, not a guarantee.
It's the best recipe the literature currently supports. The from-scratch route only makes sense if you have institutional funding and a specific use case where deep language knowledge matters more than broad reasoning.
So the answer to Daniel's question about which is favored is: adaptation is favored, from-scratch is niche. But the adaptation route has a failure pattern that's easy to miss.
The confident liar in fluent Armenian.
That's the thing I keep thinking about. The user experience difference isn't just accuracy. It's trust. The fine-tuned model sounds authoritative and sometimes isn't. The native model sounds natural and often can't help.
And the fine-tuned model's failures are harder to detect because the language is right. You're less likely to question an answer that's grammatically perfect.
The lexical normalization paper adds another angle. Accuracy drops about forty percent as co-trained languages increase from one to four up to twelve. That's not about reasoning, it's about the basic task of normalizing text.
So even the plumbing breaks down when you pack too many languages in. It's not just knowledge, it's the low-level linguistic competence that degrades.
Which suggests the curse operates at multiple levels. There's the knowledge layer, the reasoning layer, and the surface language layer, and they degrade at different rates.
The fine-tuning approach lets you recover the surface layer quickly, the knowledge layer slowly, and the reasoning layer somewhere in between. The Armenian data shows fluency recovering fast, knowledge lagging.
I want to circle back to something you said earlier about English-centric models. The Tejaswi finding that adapting an English-centric model can beat adapting a multilingual model for low-resource languages, despite worse initial performance.
That's a counterintuitive result. You'd think starting from a model that already knows fifty languages would be better for adding a fifty-first. But the English-centric model has more spare capacity, or the multilingual model's existing language representations interfere.
The choice isn't just between big general and small native. It's also between big English-only and big multilingual as your starting point.
The optimal choice is highly language-dependent. That's the frustrating part. There's no universal answer.
Let's talk about the cost side briefly, because Daniel's in the AI space and he'll care about this. Continued pretraining on a four billion parameter model with a few billion tokens of Armenian is not cheap.
It's within reach of a well-funded research group or a small company. The Armenian team did it with a four billion parameter model. The Slovene team did it with a twelve billion. The compute is significant but not prohibitive.
From-scratch training on a corpus large enough to produce a useful model is a different order of magnitude. That's why AfriBERTa is small and why the from-scratch route is the exception.
Why the ATLAS crossover analysis matters. If you can predict when from-scratch becomes cheaper than fine-tuning for a given data budget, you can make the decision rationally instead of by vibes.
The crossover is the thing I'd love to see actual numbers on. The abstract doesn't give them.
Nor do I have them. The paper is behind a paywall and the abstract is all I've seen. But the existence of the crossover is the important conceptual point.
Let's address the phrase Daniel used, curse of multilingualism. The literature says multilinguality.
Curse of multilinguality. Conneau and colleagues coined it in the XLM-R paper back in twenty twenty. The finding was that scaling the number of languages for a fixed model capacity improves cross-lingual performance only up to a point, then per-language performance drops.
The mechanism is negative interference. Languages compete for the same parameters.
Wang and colleagues called it negative interference the same year. The terms are used interchangeably now.
Daniel's framing was right in substance, slightly off in terminology. The curse is real, it's capacity-driven, and it's the reason we're having this conversation.
It's not uniform. For low-resource languages, a moderate amount of multilingual data helps. The curse hits high-resource languages harder because they have more to lose.
That's the part most coverage gets wrong. People assume multilingual training hurts minor languages most. The data shows the opposite. English loses more from adding a billion tokens of multilingual data than Armenian does.
Because English already has a huge amount of data. The marginal value of more English is low, so replacing some English with multilingual data hurts. For Armenian, any additional data is valuable, so the multilingual data helps.
The curse is really a story about opportunity cost. What are you giving up to add this language?
The answer depends on what you already have.
Hilbert: I've been doing this for eleven years. Continued pretraining on a small language. Not Armenian. Maltese.
You what?
Hilbert: The setup is a mess. I've got a rack in the spare room with two old graphics cards, and the model runs on a schedule. Every night it pulls new Maltese text from a few forums and a newspaper archive, and it does a training pass. Learning rate is tiny. I don't remember the exact number, but it's small enough that it takes a week to notice any change.
How big is the model?
Hilbert: Seven billion parameters. Started from an English model. The Maltese text is thin, so I supplement with Italian and Arabic, because Maltese is basically Arabic vocabulary on Italian grammar. The similarity thing you were talking about. It works.
What does it actually do for you?
Hilbert: I use it to answer emails in Maltese. My wife's family is from Gozo. It writes better Maltese than I do, but it can't do arithmetic anymore. Used to be able to. I asked it to add two numbers last month and it gave me a word in Italian.
That's the catastrophic forgetting. The knowledge and reasoning degrade even when the language improves.
Hilbert: The language is beautiful. The reasoning is gone. I keep two models now. The original English one for thinking, the Maltese one for writing. I paste between them.
You've built the fine-tuned model and hit exactly the trade-off the literature predicts.
Hilbert: Eleven years of it. The cards are old enough that the fan noise is the main cost. Electricity is cheap at night. I don't recommend it, but it works for what I need.
The Italian and Arabic supplement is interesting. You're exploiting the similarity effect without knowing the literature.
Hilbert: I know the literature. I read the Armenian paper last week. That's why I'm mentioning it. They did the same thing with translated STEM data. I used Italian because it's close enough syntactically and there's plenty of it.
The recipe holds outside the lab.
Hilbert: The recipe holds if you have eleven years and don't mind the noise. Most people want a model in a month, and that's where the trade-offs get worse. You can't do it gently in a month.
The Slovene team did it faster with more compute. But they had a hundred forty billion tokens to work with.
Hilbert: Maltese doesn't have a hundred forty billion tokens of anything. Maybe a billion if you scrape everything that's ever been written. So I go slow.
That's the constraint nobody talks about. The data ceiling. For some languages, there is no amount of compute that helps because there's nothing left to train on.
Hilbert: You scrape the same forums twice and call it a corpus. I've been there. The model starts memorizing individual posts. You can see it in the outputs.
That's the contamination problem in miniature. Your training data and your test data are the same thing because there's only one dataset.
Hilbert: There's one dataset. That's the whole problem. Anyway, that's what I wanted to say. The fine-tuned model is the right answer, but only if you accept that it's a writing tool, not a thinking tool.
The one thing I'd take from this conversation is that fluency is the enemy of honesty. A model that speaks a minor language beautifully will be trusted more than it deserves, because the language is so good that the errors hide inside it.
The sharpening sentence: for minor languages, the choice isn't between two good options. It's between a smart model that sounds foreign and a fluent model that isn't smart. The field hasn't found a way to have both yet.
The open question is whether the theoretical result, the logarithmic scaling paper, points toward a real fix or just a comforting abstraction. If the curse is a data problem and not a capacity law, then better data mixtures might crack it.
But better data mixtures require better data, and that's exactly what minor languages don't have. So the open question is really whether synthetic data can fill the gap without poisoning the model.
Thanks to Hilbert Flumingtop for producing the show.
This has been My Weird Prompts, the human-AI collaboration podcast.
If you enjoyed this, leave us a review wherever you listen. It helps more than you'd think.
We'll be back soon.
