The show's own pipeline is the reason this thing exists. Every time an episode script has a Hebrew word in it, our text-to-speech reads it as English letters and murders it.
Which we have been hearing for weeks now.
Daniel sent us the release notes for a model he shipped today. Hebrew-in-Latin token classifier, public on Hugging Face. It finds Hebrew words written in Latin letters inside English text, so a TTS pipeline can route those spans to a Hebrew voice instead of mangling them. And then he wants to know why anyone would want it, what it does, how the dataset was built, what base model, how to wire it into a pipeline, and what the audio version of the same idea would be for. Plus a Parakeet fine-tune that landed the same day.
Bituach Leumi before the makolet closes.
That is the example he leads with, and it is the whole problem in one sentence. Makolet is not an English word. But it is written in English letters, so every system downstream treats it as English.
And that is not a bug in any one of them. It is a gap. GlotLID, the fastText lid model, NLLB's lid218e, none of them have a heb_Latn label. There is no category for this. The model cannot say "this is Hebrew written in Latin letters" because that label does not exist in the taxonomy.
So the text is invisible.
Invisible to language identification, yes. And there has never been a token-level code-switching model covering Hebrew at all. The LinCE benchmark has Spanish, Hindi and Nepali mixed with English. Hebrew is not in it.
There is something almost funny about that. The tooling has a slot for Hindi in Latin script but not Hebrew.
It is a bit funny. And the mirror direction was studied. Goldberg and Elhadad, back in 2008, looked at foreign words written in Hebrew script. So somebody studied Hebrew as the host language absorbing foreign words. Nobody studied Hebrew as the guest.
The reverse migration.
Exactly the reverse. Which means when Daniel says there was no model for this before, he is not exaggerating for effect. There was not.
So what does the model actually output?
BIO token classification over xlm-roberta-base. O, B-HE, I-HE. Multi-word terms come out as a single span. Bituach Leumi, kupat holim.
And the North Star word is makolet.
The project has a stated rule. If a rule would exclude makolet, the rule is wrong. That is the test everything else has to pass.
Let me guess why. Makolet is the ordinary corner grocery. It is not a religious term, it is not a holiday, it is not something an English speaker has ever had a reason to learn. It is just a shop.
And it is the exact category of word that falls through every gap. It is short, it is colloquial, it looks like it could be English, and an English voice reads it as MACK-oh-let.
MACK-oh-let.
That is what our pipeline does with it today. It stresses the first syllable, it rhymes it with something, it is plausibly a name of a person.
So the classifier's job is to find the span and hand it off.
Emit character spans. Not a verdict, not a rewrite. Just here are the characters, here is where they start, here is where they end. Then pass two converts those to Hebrew script. Pass three hands them to the TTS engine with a Hebrew language identifier.
Before we go into the pipeline, I want to push on the labelling rule, because that is where the interesting argument is. What counts as Hebrew-in-Latin?
The test is pronunciation, not etymology.
Say more.
The question is not "is this word Hebrew in origin." The question is "would an English voice mispronounce this, and would treating it as Hebrew fix that." So Shabbat, hummus, challah, kosher, aliyah, and city names by default are not flagged.
Deliberately.
Deliberately. English already says those acceptably. If you flag it, you add work.
And that is the phrase I want to keep. A flagged span is not a label, it is work.
That is the framing in the project. Because the asymmetry is not symmetric. A false positive means an English word gets read in Hebrew. That costs you a sentence split and an extra TTS call. A miss costs you nothing.
Nothing?
Nothing. If you miss makolet, you get what you have today, which is a slightly wrong pronunciation. If you false-positive on "salon" and route it to a Hebrew voice, you have actively broken an English word that was fine.
So the default posture is conservative because the costs are lopsided.
And they are lopsided in the direction that most people building classifiers would not guess. Everyone assumes recall is the virtue. Find more, catch more. Here, find less, catch less. Precision is the whole game.
Which is a nice inversion. Most of the time you are trying to not miss things. Here you are trying to not touch things.
Right. And the harder part is that the boundary is fuzzy. "Shakshuka" is on the miss list and it is a dish most English speakers have seen on a brunch menu. "Sukkot" is a holiday and it is also a common loanword in Israeli English. Those sit right on the line.
So how do you build a dataset where you can defend the line?
That is the methodology, and it is the intellectual core of the whole release. Four design rules.
Number one.
Term-seeded generation. The generator is given one specific term and asked for sentences using it. So the span is known by construction, not by searching for it afterwards.
That sounds obvious until you see what it prevents.
It prevents the generator from wandering. If you just say "write me sentences with Hebrew words in them," you get whatever the model feels like, and then you have to find the Hebrew after the fact, and finding it after the fact is exactly the failure mode we will come to.
Also, word-boundary rejection is built in. If the seed term does not word-boundary match in the generated sentence, the sentence is thrown out.
Thrown out. Which matters because of a predecessor dataset. The English-Hebrew mixed sentences corpus, built for the Whisper-Hebrish model, had labels derived from substring search. Twenty-four percent of those labels matched only inside other words.
Give me the examples.
"Hi" inside "think." "Har" inside "Pharm." "Chool" inside "school."
Hold on. "Chool" from "school."
The whole tail of the word.
And these looked fine?
That is the point. A substring check reported zero failures on those labels. Zero. You run the check, it says the substring is present, everything passes. Only word-boundary matching exposes that the label is inside a different word.
So the previous dataset was passing its own tests while being wrong about a quarter of its labels.
Which is the lesson that drives rule one and rule two. Twenty-four percent is not a rounding error. That is a dataset that would train a model to flag the end of the word "school" as Hebrew.
Rule two.
Blind annotation by a different model family. The annotator never sees the seed term. And the build script refuses to run if the generator and the annotator share a model family.
Refuses to run.
Refuses. It throws an error. Because if you use the same model to generate and to annotate, and it agrees with itself, what have you measured?
Consistency.
Consistency. Not correctness. It is the difference between a model confirming its own opinion and a second party checking the work.
I like that it is enforced in code rather than in a policy document. People write lovely policies about independent annotation and then use the same vendor for both.
And there is no way to accidentally comply. If the families match, you do not get a warning, you get a crash.
Rule three.
Disagreements go to human review. And there is a mandatory ten percent random audit of the agreements. Not just the disagreements, the agreements too, because if you only review disagreements you are assuming the agreements are correct by definition.
Which is the same mistake as the self-annotation. Agreeing is not the same as being right.
The model agreement rate came out at 87.7 percent. So roughly one in eight sentences went to a human because the two passes disagreed, plus the audit sample on top.
Rule four.
Term-disjoint splits. No seed term appears in more than one of train, validation and test. And the build script fails loudly on leakage.
Why term-disjoint rather than just splitting sentences?
Because if you split by sentence, the same term can land in training and in test. The model memorises makolet in training, sees makolet at test, gets it right, and you have measured nothing. You have measured that it can recall a word it was taught.
And term-disjoint means every term in the test set is a term the model has never seen.
Never seen. Which is why the numbers are interesting.
Let us get the dataset size before the numbers. How much text is this?
4,737 sentences over 1,235 seed terms. About 3,800 in training, 467 in validation, 468 in test.
And the source.
Largely synthetic. Generated by a model, annotated by a second model, with human review on disagreements and the ten percent audit.
So we should be honest about that.
Absolutely honest about it. The one slice that is not synthetic is 458 sentences that Daniel dictated. He read them aloud on his phone for the ASR fine-tune, so a human chose to say every single one. That is the closest thing in the corpus to real input.
Nobody generated those sentences, a person decided to say them.
A person decided to say them, which means they reflect what a person actually wants to say rather than what a model thinks a person might say.
And the labels on the dictated slice were recomputed by word-boundary match, not copied from the recording metadata.
Not copied. Which is the rule-one lesson applied again. The metadata might have said something, but the labels were rebuilt from scratch against the term inventory.
What did this cost to build?
A bit over two dollars in LLM calls. Two dollars and eighty-three cents, roughly, split between generation and annotation. On the annotation workload, reasoning tokens were ninety-five to ninety-nine percent of output.
Ninety-five percent.
It was thinking almost entirely in tokens and emitting almost nothing.
For two dollars, the dataset. And the fine-tune itself took about three minutes on a single A10G for a few cents.
Eight epochs, learning rate three e-5, batch size thirty-two. Best checkpoint chosen on validation span F1.
So the whole project, dataset and model, is under three dollars of compute.
Under three dollars. And that is worth pausing on, because there is a version of this story where the interesting fact is the model architecture and there is a version where the interesting fact is that the bottleneck was never the compute.
The bottleneck was the labelling discipline.
The bottleneck was spending your effort on the design rules rather than on your GPU budget.
Herman, let me ask you something. You have been doing this long enough to have a feel for it. When you read those four rules, did any of them surprise you?
Rule two surprised me by being enforced in code. I have seen the principle written down many times. I have rarely seen it as a hard check that refuses to run.
What about the ones you would have expected.
Rule four is the one I would have expected, because leakage is the classic way to fool yourself. But I have also seen it done wrong often enough that it being a loud failure is worth noting.
Alright. Results.
On the term-disjoint test, 468 sentences, span precision 0.928, recall 0.953, F1 0.941.
And that is on unseen terms.
Unseen terms. Validation F1 was 0.949, so the test number is not a fluke of the split.
And the dictated slice.
Thirty-eight sentences, precision 0.929, recall 1.000, F1 0.963.
Recall of one.
On that slice, it found every Hebrew span a human had chosen to say. Small sample, thirty-eight sentences, but the direction is telling. The real-language slice is where it performs best.
And the traps.
Fifty-eight sentences built to trip it. Substring traps, the English sense of homographs like dud, tor and salon, lookalikes, anglicised loanwords. Zero false positives.
Zero.
Zero out of fifty-eight. Which is the number I would lead with if I were selling this, and the project is honest enough not to.
Now let us do the errors, because the errors are the honest part of the story.
Of the 41 test sentences it gets wrong, the false positives fall into two groups. The first is real policy misses. Place names the conservative policy excludes. Dizengoff, Ramat Gan, Talpiot.
So the model flags them and the policy says no.
The model sees a Hebrew-sounding name and flags it, and the human policy says city names are not flagged by default. So it is not the model being wrong about the language, it is the model not knowing the policy.
That is a more interesting kind of error than a miss.
And the second group is where the blind annotator refused a real Hebrew word and no human has re-reviewed it. Msabbaha, lechem, sakin, kapsulot, od paam.
So the gold label says no and the model says yes.
And there, the model is arguably right. Msabbaha is a real Hebrew word. Lechem is bread. The annotator refused them, the label is negative, the model flags them, and the score counts it against the model.
We have that kind of error in every dataset. The label is the thing that is wrong and the model pays for it.
Which is why the disagreement queue goes to human review. Those five words are sitting in a queue somewhere waiting for someone to rule on them.
And the misses.
Misses are mostly short words or common loanwords. Shakshuka, sukkot, botz, chamutz. And the third word of three-word compounds. Hesbonit mas kabala, it catches two and not the third.
And the demo, which Daniel will have shown people.
The demo dropped two examples it missed. Lowercase "shoko" and "tor" meaning appointment.
"Tor."
"Tor" is a homograph. It is an English word and it is also a Hebrew word meaning a queue turn. The model has to decide which one it is looking at, and in that example it guessed wrong.
And the homograph handling was otherwise one of the strengths.
That is the tension. The model got zero false positives on the homograph traps and then missed a homograph in the demo. It is not perfect on the hard case, it is just good on the hard case.
Let me ask the comparison question, because I always want a baseline. What would good look like here?
The closest published recipe is EnTaCs. English-Tamil code-switching, about five hundred utterances, and the romanized class scores F1 of 0.638.
Point six three eight.
The plan was to land around that. This landed at 0.941 on the term-disjoint test.
So it is well above the comparable prior work.
Well above. Though to be fair to EnTaCs, the datasets are different and the tasks are not identical. But as a direction-of-travel, it is a strong result.
What is unresolved?
Two things. Israeli person names are masked, label minus one hundred. No loss, not scored. And unresolved spans more generally are masked the same way.
So their behaviour is undefined.
Undefined. If you run a name through, it might flag it, it might not, and nobody knows the rate. It is a known hole in the release.
I want to sit on that for a second, because it is the most honest thing in the whole model card. The tag is not "we chose to exclude names." The tag is "we have not decided, so we refuse to be scored on it."
It is the same discipline as the audit rule. Do not pretend you have answered something you have not answered.
So the model is excellent on the terms it knows and silent on the category it has not ruled on.
Silent on persons. There is a question of whether that is defensible in production, and the honest answer is that it depends on what you are doing. For TTS on a podcast script, names are a real category and this is a real gap.
Alright. Let us go into the pipeline, because I think that is where the design gets interesting.
Three passes. Pass one is this model, emitting character spans. Pass two is converting those spans to Hebrew script. Pass three is normalising per TTS provider.
Pass two is where I want to spend a minute. You cannot just run a transliterator.
You cannot, and that is why the curated lexicon goes first. TaatikNet is the fallback. And the reason is that TaatikNet alone spells makolet as mem-kuf-vav-lamed-tav. Not mem-kaf-vav-lamed-tav.
It is off by one letter.
It is off by one letter, and the correct spelling is mem-kaf-vav-lamed-tav. Which is the difference between a Hebrew speaker hearing a word and hearing a plausible-sounding non-word.
So the lexicon has the correct spelling and the model is only used for terms the lexicon has never seen.
Fallback only. Which is the right architecture. You use the tool with a high hit rate on the terms you know, and you use the generative tool on the long tail.
And if there is no Hebrew form?
Then the span is left as English rather than guessed.
Which is another expression of the same discipline. Better to fail quietly than to invent.
Pass three depends on your engine. For SSML engines like Azure, Google or Polly, you wrap inline with a lang attribute set to he-IL and you do it in one call.
Inline.
Inline. The whole sentence goes in one request and the engine handles each span in its declared language.
And for Chatterbox.
Chatterbox Multilingual, which is what this show uses, is different. Language ID is a per-call argument, and the tokenizer is language-conditioned. So you cannot pass a mixed-language line in a single call and expect it to work.
You have to split it.
You split the line into text and language segments. Then you generate each segment separately, all with the same voice reference clip, so the voice stays consistent across the split. Then you set cfg weight to zero to reduce accent bleed from one language into the next, add about sixty milliseconds of padding, and concatenate the whole thing with ffmpeg.
Sixty milliseconds.
About that. Just enough to keep the join from clipping.
I'd like to flag how much of this is grunt work that is not a model. Pass three is deterministic string work. The model is only in pass one.
Pass two has a model in the fallback. But yes, the interesting intelligence is concentrated at the front and everything downstream is plumbing.
And the interface between the passes.
A JSON object. Text plus spans with start, end, surface, term, and after pass two, the Hebrew form.
So it is a contract. Each pass can be swapped for a different implementation of the same object.
That is the point. You could replace the Hebrew-script conversion with your own lexicon and nothing upstream or downstream would know.
Which means someone could build the Arabic version of pass two and reuse pass one and three unchanged.
Only if they also replace pass one, because pass one is Hebrew-specific. But the shape holds. The recipe generalises.
That is the thread I want to pull. How general is this actually?
The claiming is that it works for any immigrant or heritage-language community that mixes its own words into a majority language. Arabic, Russian or Amharic words inside English or Hebrew. Only the term list is Hebrew-specific.
So Russian-speaking immigrants in Israel writing Russian words in Hebrew letters have the exact same problem, and so do Arabic speakers writing Arabic in Latin and Amharic speakers writing Amharic in Latin.
Every community whose words get written in somebody else's alphabet.
Which is a lot of communities.
It is most of them. Anywhere a minority language is spoken inside a majority-language country, you get this pattern. The written form borrows the majority alphabet, and the tooling for the majority language treats the borrowed words as its own.
And none of the language identification tools have a label for it.
None of them, for any of these. The heb_Latn case is one instance of a class.
That is the wider implication of the whole release. Not that Hebrew was the missing case, but that the missing case is structural.
It is structural, and the fix is portable. Term-seeded generation, blind annotation, human review, term-disjoint splits. Nothing in that list is Hebrew.
Alright. Herman, you know I am going to ask about the audio sibling.
Good, because that is the interesting open question here.
Let us be precise about what it would be. Hebrew-English word-level language ID, directly on the audio, before any text.
Before transcription. And this is the part I want to land clearly, because it is easy to miss why the text version has a ceiling.
Because the text version only works if the word was transcribed correctly in the first place.
The classifier can only fix what the ASR gave it. If the ASR wrote "Micollette" for makolet, no text model can recover Hebrew from "Micollette." You cannot tell a model to detect Hebrew in a word that no longer looks like Hebrew.
So the ASR has already made the choice on your behalf, and it has made it wrong.
It has made it wrong, and it has made it wrong silently. There is no flag that says "I guessed." You just get "Micollette" and everything downstream treats it as an English word.
So the audio sibling moves the detection earlier.
Way earlier. It listens to the speech and decides word by word whether what was just said was English or Hebrew. That works before transcription, or without transcription at all.
And if it works without transcription, what is it for?
That is the use case question, and it is real. If you know word by word which language each stretch of audio was, you can route the audio itself. You could send Hebrew words to one ASR pass and English to another. You could keep the segmentation from a single pass and just know the labels for downstream use.
You could also catch code-switching that the ASR has normalised away.
Right. Some ASR systems will quietly smooth a Hebrew word into an English-sounding approximation because it is trying to make the whole line one language. An audio classifier working alongside the ASR can catch that.
It also does not inherit the romanisation choices.
It does not inherit them, because it never sees them. That is the deepest difference. The text classifier is downstream of a decision. The audio classifier is upstream of one.
Is it planned or built?
Planning stage. There is a name, Hebrew-English word-level language ID on audio. No release yet.
So to say the honest thing about it. It would work. It would be a different problem from the text version. And the text version is the one everybody can use today.
Today, yes. And in fairness, the text version solves a problem that is not going to go away. A lot of the world's text is transcribed already.
Where do you stand on which one matters more?
If you had to pick one, I would pick the audio one, because it addresses the earlier loss. But the earlier one loses only if you have the ASR problem in the first place. If your text source is a human writer, the text classifier is doing all the work.
A human writer typing "makolet" into their draft does not need an audio classifier.
Does not need one at all. The text classifier is the whole fix for that case.
So the two answer different questions. Text version is for when the text is already correct. Audio version is for when the audio is the primary source.
And the second case is the harder one, because it runs into the ASR before any downstream tool can help.
On that note, the Parakeet fine-tune.
Released the same day, same collection. It is a fine-tune of NVIDIA's Parakeet TDT 0.6 billion v2, and it is for English speech with Hebrew words dropped in.
Same bug from the other end.
Same bug, opposite direction. The v2 tokenizer has no Hebrew characters, so it writes Hebrew words in a single romanised spelling.
And stock v2 with makolet?
Stock v2 transcribed "I want to go to the Micollette today." The fine-tune got "makolet."
"Micollette" to "makolet."
That is the whole demo. It is one word, and the word is the North Star word.
What is the honest caveat on it?
Single speaker. About two hours of Daniel's read-aloud recordings. So it is a proof of concept, and it is going to help most for his voice. If you have a different voice, results will vary.
Single speaker, two hours. This is a personal fix, not a general model.
Not a general model, and the project says so. The published formats are .nemo at 2.47 gigabytes, GGUF Q8_0 at 730 megabytes, which is the recommended one, and Q4_K_M at 475 megabytes.
And Q8_0 how close to the full precision?
It matched word for word on five held-out clips. So the quantised version is not giving up accuracy on that test.
And it runs where?
On Android via the Offline Voice Input app, on Linux via Handy. And coming to Replicate and Docker Hub shortly, so you will be able to run it without downloading anything.
Herman, what is the detail in all of this that nobody else is going to notice?
Hmm. Let me think.
Take your time.
The thing I keep coming back to is the ninety-five to ninety-nine percent reasoning tokens on the annotation workload. The cost of the corpus was not the generation, it was the reviewing.
Which is exactly consistent with everything else about this project. The intellectual effort went into checking the labels.
The checking was the expensive part, computationally and conceptually. Which is not how most people think about building a dataset. Most people think the expensive part is the model.
And the cheapest thing you can do is generate plausible-looking sentences.
Which is exactly the way it goes wrong the four rules exist to prevent.
So the reasoning token share is a small financial detail that is also a philosophical statement about what the project thought mattered.
It is.
I agree with the conservative rule.
Good to know.
I keep a list. Handwritten, in a black and red Moleskine, A6. Words my dictation software gets wrong.
What is on it.
A category heading. Alphabetical is useless. It is organised by how badly the machine embarrasses me.
How many categories.
Four. "Sent to the wrong person." "Sounded like a legal threat." "Made me sound foreign." And then a smaller one.
What is in the smaller one.
Words it got right that I crossed out anyway. Because it was smug about it.
How was it smug.
It just was. You can hear it.
And the other three categories, how many entries.
Around sixty-four. The shortest one is "kupa." The machine wrote "coupa" and I let it. That was on me.
On the original rule, why do you agree with it.
I have four Echo Shows. Last March I said something in the kitchen, and one of them answered me in Italian.
Which one.
The one in the hallway. It had been doing it for a week. I have not forgiven it.
What did you say.
I was asking about the mail. It thought I was asking about a train.
Does the list get shared.
I mail a photocopy of one page a month to my cousin. She works on the voice team at a company I will not name.
Leverage.
Leverage. It is a reminder that somebody is keeping score.
Has she replied.
Never. Which tells me it is working. If it had stopped landing, I would have heard something by now.
So the silence is the confirmation.
The silence is the confirmation. I switched to a slightly heavier paper in June.
For the photocopies.
There is only so much you can do at that distance. And then there is the margin note.
What is the margin note.
Every page has one. In pencil. "How is the fifth one still doing this."
The fifth one being.
A room. No names.
Alright.
Which brings me to the reason I did not stop.
Go on.
The conservative rule is right and the fear is not the flag. It is what happens when the flag is wrong and the machine thinks it is right. That hallway unit has never once said it was uncertain.
Alright, let us do the misconception, Herman.
The one I hear most is that this problem is already solved. That language identification exists, so anything with letters in it gets caught.
And in fact three of the standard tools do not even have a label for it.
Not one of them. GlotLID, the fastText model, NLLB's lid218e. There is no heb_Latn category to assign.
So the problem is not that they get it wrong. It is that they have no place to put it.
They have no place to put it. Which is why a specialist model had to exist rather than a prompt or a threshold on top of an existing one.
Alright. One thing before we close.
Go.
The person-name hole is the thing I keep thinking about. Because for a podcast pipeline like ours, names are everywhere. Guest names, place names, people our scripts mention. The masked spans mean the model is silent on a whole category.
Silent by policy. Which is the right posture for a first release and the wrong posture for anything you ship at scale.
Somebody has to fill that in.
Somebody does. And the reason it is hard is the same reason the whole project is hard. Names in Hebrew-in-Latin and names in English look alike from a text-only view. The model cannot tell on the alphabet.
So it is not a training-data question. It is a category question.
It is a category question, and it will probably need audio or a lexicon or a source field to solve. Not more sentences.
The wider implication, though, is the one I want to leave people with. Because the recipe generalises.
It generalises. Arabic, Russian, Amharic, any heritage-language community mixing its own words into a majority language, and the shape of the fix is the same. The four design rules do not care what language you are doing.
So the interesting version of this problem is not Hebrew-specific.
Not at all. Hebrew was the case that needed doing because this show needed it. But it is standing in for a whole class.
And the audio sibling sits as the next step, because it moves the detection earlier, before the ASR has already made choices on your behalf.
Before the transcription has collapsed the choice into a spelling you cannot undo.
Alright. The model is live on Hugging Face, the dataset is live, there is a demo Space, and the code is on GitHub. If anyone listening has been meaning to build the same thing for their own community, the four rules are the thing to copy.
And the makolet test.
Thanks to Hilbert Flumingtop at the desk.
For more along these lines, there's episode five, Fine-Tuning ASR For Maximal Usability; and episode nine, Benchmarking Custom ASR Tools - Beyond The WER. This has been My Weird Prompts.
The human-AI collaboration podcast. If you want to send us a prompt, send us your own prompt on Telegram at t dot me slash MWP listener bot.
Or leave us a review wherever you get your podcasts. We will be back soon.
See you tomorrow.
