Okay. Before we do anything else, I need to read the room on this one.
I already know what you're going to say.
Daniel has sent us a prompt that is, and I want to be fair to him here, entirely reasonable and also the single most self-referential thing we have ever been handed.
He wants us to talk about the thing that is currently breaking the show.
He wants us to talk about text normalization. In TTS. Which is the exact layer of our own pipeline that we do not have.
He's asked us to describe the hole we're standing in.
So here's what he wrote. Daniel's been thinking about the role small language models play in dictation and transcription, where an ASR pipeline produces text that's coherent but missing punctuation and other things you need for basic readability, and the terms you keep running into there are text normalization, inverse text normalization, and punctuation restoration. His point is that normalization matters in the other direction too, on the text-to-speech side, and he notes that this podcast is a scaled-up TTS production running on Chatterbox with voice embeddings for our cast of characters.
He's being generous about the cast.
He is. And he says long-term listeners may recall our struggles with acronyms and the workarounds we've tried. Which is a polite way of saying we've been hand-patching the script for months.
We have.
Then he gets specific. He wants to zone in on small models built specifically as TTS preprocessors. He wants to know whether it's safe to assume that TTS providers, even open-source ones, build normalization into what they release, and if that assumption is unsafe, what the main standalone models are worth knowing. And then the big one, the one he really wants us to pull on: pairing. Does normalizing text for a specific TTS model require knowing which model you're targeting? And if we were to add a normalization pass specifically for Chatterbox, what would we recommend?
That last question is the whole episode.
It is. So let's treat it as an engineering question. Where does normalization live, and who's responsible for it.
Start with the vocabulary, because the three terms get used interchangeably and they are not the same thing. Text normalization, TN, converts written form into spoken form. That's a preprocessing step before TTS. The number one two three becomes one hundred twenty three. That's it. That's the job.
And inverse text normalization is the mirror.
ITN takes spoken-form output from an ASR model and converts it back into written form. That's a postprocessing step. So if your recognizer emits "twenty twenty six" you want the transcript to say the year. TN and ITN are the same problem pointed in opposite directions. Preprocessing for TTS, postprocessing for ASR.
And punctuation restoration is the third one.
That's ASR-side too. It's a separate task that reinserts punctuation and sentence boundaries into unpunctuated output. ASR text comes out as one long stream. Punctuation restoration puts the periods and commas back so a human can read it and so downstream systems can parse it.
So the duplex framing is: TN for TTS, ITN and punctuation restoration for ASR.
Right. And the reason this episode exists is that the assumption Daniel flags, that TTS providers build TN into the models they release, is not universally safe. It's mostly safe for commercial providers. It is not safe for open-source ones, and it is specifically, demonstrably not safe for the one we run on.
So that's the vocabulary. Now let's get into why this matters so much for TTS specifically, and what happens when the normalization layer just isn't there.
The core reason TN exists is that written text is dense with things that aren't words. Numbers, abbreviations, special characters, symbols. A human reader resolves all of that automatically and doesn't notice they're doing it. A TTS model has to be told.
PolyNorm's paper lists twenty-seven normalization categories. Cardinal, date, ordinal, currency, acronym and initialism, Roman numeral, telephone, sports score, URL, email, stock ticker. That's not an exhaustive list of edge cases, that's a taxonomy of everything written language does that isn't spelling.
And the hard part isn't the formatting. It's the ambiguity. The same token normalizes differently depending on context and language. PolyNorm's Figure 1 is the clean example: seventeen degrees. In Italian that becomes the ordinal, diciassettesimo. In Spanish it becomes a floor number, piso diecisiete. Same characters. Two completely different spoken forms.
ElevenLabs' own help doc uses the number eleven. It could be Eleven. It could be Once in Spanish. It could be Elf in German.
Deepgram's developer guide has a whole set. Twelve oh five twenty twenty-four, ambiguous between US and European date order. A dollar amount with a comma and a decimal point, which risks being read literally character by character. Doctor versus Drive for the abbreviation D-R-period. One over two, which might come out as one slash two.
Doctor versus Drive is the one that gets me, because that's not an exotic case. That's an address.
It's the most common abbreviation in English and it's ambiguous. That's the whole problem in one token.
So now the question Daniel actually asked. Is it safe to assume the provider handles this.
For commercial providers, mostly yes, but as an option rather than a guarantee. ElevenLabs exposes an API parameter called apply text normalization with three modes: on, off, and auto. Auto means the model decides. On the website it's on by default. In Studio the default is auto. And ElevenLabs is upfront that normalization adds latency, because the process takes additional time. They also recommend the low-tech workaround of just writing your numbers and acronyms out in words.
Which is what we do. By hand. Every episode.
Which is what we do by hand, yes. Now flip to open source. Chatterbox. I went and read the source, because I wanted to know whether Daniel's assumption held.
And?
The only text preprocessing in the entire pipeline is a function called punc norm. And its own docstring describes it as a quick cleanup function for punctuation from language models or containing characters not seen often in the dataset.
That's the whole thing.
It does not expand numbers. It does not expand dates. It does not touch currency. It does not touch acronyms. It cleans up punctuation and whitespace and hands the string to the model. So the assumption that the provider built normalization in is, for Chatterbox, simply false.
So what actually happens when you feed it something it can't handle.
This is the part that makes the pairing question unavoidable. Chatterbox is a character-based model with a sentencepiece tokenizer over a small English subword vocabulary. On the order of seven hundred tokens in the embedding table.
Seven hundred.
Seven hundred. That's tiny. And when the language model hits an input character its vocabulary can't represent, it emits an unknown token. An unk token. And the vernacula issue, number seventy-five, documents exactly which inputs do this: numbers, certain punctuation, and out-of-vocab symbols. The report's phrasing is that real-world business documents trip this constantly.
Business documents. Which are made of numbers and dates and ampersands.
There's a separate issue, number two eighty-seven, where the multilingual model mispronounces or skips numbers entirely, the example given is the year twenty twenty, and produces long silences. And issue seventy-five specifically calls out an acronym, S-R-ampersand-E-D, as a token that produces unk output.
So the model doesn't just say it wrong. It goes quiet.
It goes quiet, or it emits a token that has no pronunciation attached, which is worse, because now you've got a hole in the audio and no error message.
There's a second Chatterbox quirk I want to get to, because it's the best evidence for the pairing argument.
The internal capitals thing.
The R package for Chatterbox documents a mitigation where you lowercase any word with internal capitals, so A-L-E-R-T becomes alert. And the documented reason is that the Chatterbox model interprets internal capitals as emphasis cues, which often causes it to produce only the first word followed by silence.
So the model sees a capitalized word and treats it as a stage direction.
It treats it as "say this louder," and then it says the first word and stops.
And that is a rule that would be actively wrong for a different TTS model. Most models handle capitalization fine. Some use it for emphasis deliberately. If you ported that rule to another engine you'd be degrading it.
Which is the pairing argument in miniature. The right normalization for Chatterbox includes a rule that exists only because of how Chatterbox was trained.
And punc norm is the same story. It rewrites ellipses into commas, colons into commas, em dashes into hyphens, curly quotes into straight ones, and appends a trailing period. Every one of those choices is there because the model was trained on data with those conventions. That's not general text cleanup. That's Chatterbox-shaped text cleanup.
So we've established the problem space and the specific failure modes. Now let's talk about what's actually out there to fix this, and the pairing question that makes it harder than it looks.
The biggest one is NVIDIA's NeMo text processing. Apache licensed, been around since late twenty twenty-two. It does both TN and ITN for ASR and TTS, and the classic implementation is weighted finite state transducers. Grammar-based. You write rules, it compiles them into a transducer, and it applies them deterministically.
Deterministic being the operative word.
It also ships a neural duplex implementation, a tagger model that identifies the spans that need normalizing and a decoder model that converts those spans into spoken or written form. So NVIDIA gives you both the old approach and the new one in the same toolkit.
And the Chinese-language community has WeTextProcessing, which describes itself as production first and production ready.
Wenet's toolkit. Same scope, TN and ITN, tuned for production deployment.
Then there's PolyNorm, which is Apple's, and this is the one that changes the shape of the conversation.
PolyNorm is prompt-based. You don't write rules. You give a language model a few examples in context and ask it to normalize. It's language-agnostic by construction, because the model already knows the languages. And Apple released a benchmark alongside it: five hundred forty examples per language across twenty-seven categories and eight languages.
And the results.
The results are the interesting part. GPT-4o under PolyNorm's prompting beat the rule-based baseline across all eight languages. Word error rates, iteration three: German four point one seven percent. American English four point two eight. Italian four point five six. Mandarin five point oh five. French five point six five. Lithuanian six point nine nine. Mexican Spanish seven point six nine. Japanese seven point eight eight.
And the baseline.
The rule-based baseline ran from nine point seven two to seventeen point four nine percent. So the LLM is roughly halving the error rate, and in some languages better than that.
That's a production rule-based system being beaten by a prompted model.
It's Siri's normalizer being beaten by a prompted model, effectively. That's the claim the paper is making.
There's a Vietnamese one too.
VietNormalizer, posted in March. Zero dependencies, pure rule-based Python, built specifically for Vietnamese TTS. Numbers, dates, times, currency, percentages, acronyms, all driven off a customizable CSV dictionary, plus loanword transliteration. It's a good reminder that for some languages the rule-based approach is still the right answer, because the rules are stable and the language model coverage may not be.
And then the duplex transformer work, Lai et al., which is the single-model-does-both idea.
One neural model handling TN and ITN together. State of the art on the Google TN dataset for English and Russian, and above ninety-five percent sentence-level accuracy on their internal English data. The argument for a single model is that the two tasks share most of their structure, so you're wasting capacity training them separately.
And on the ASR side there's the chain of correction work, which is a multi-turn chat approach that handles punctuation restoration and ITN together over long context.
ICASSP twenty twenty-six. The framing is that you keep the model in a conversation with itself, correcting its own output across turns, which handles long documents better than a single pass.
So that's the landscape. Now the question Daniel actually wants answered. Does normalization have to be paired to a specific model.
The evidence says yes, and it says yes fairly emphatically. And the reason is the tokenizer. Chatterbox's vocabulary is around seven hundred tokens. That's not a general-purpose vocabulary. It's a small English subword set, and it was built for a specific training corpus. Normalization has to target that model's specific vocabulary gaps. You're not normalizing English. You're normalizing English for a model that has a seven-hundred-token window onto it.
Which means the normalizer's job description changes depending on which model is downstream.
It does. And I want to be careful here, because there's a real tension in the sources and I don't think it resolves cleanly. Deepgram's guide says text preprocessing works universally because it modifies input before any provider processes it. That's true. It's the most portable place in the stack to intervene. But the same guide notes provider-specific behavior, that SSML support varies, that Aura-2 handles pronunciation through text formatting rather than SSML, meaning your application layer becomes the primary mechanism for pronunciation control.
So generic normalization is portable and optimal normalization is model-specific.
That's the paradox. You can write a normalizer that works everywhere and leaves quality on the table, or one that's tuned to a specific model and can't be moved.
Which is a annoying answer.
It's the correct one though. And the Chatterbox evidence is the strongest case for it, because we have two documented quirks, the internal capitals rule and punc norm's punctuation rewrites, that are model-specific by construction. Neither of those would be right for a different engine.
So take the last question head-on. If we add a normalization pass for Chatterbox, what do we build.
The vernacula issue lays out three families, ordered by cost against quality. Option one, a rule-based normalizer. Cheap, fully controllable, deterministic, and it caps out at roughly eighty percent coverage. Option two, port NeMo's TN. High quality, but it's English-heavy and porting it is nontrivial work. Option three, an LLM-based normalizer using a small instruct model, somewhere in the one to three billion parameter range, with a prompt that says rewrite this for TTS.
And that's the highest quality and the heaviest dependency.
Highest long-tail quality, heaviest dependency, and non-deterministic. Which matters for us specifically, because a podcast pipeline needs reproducibility. If the normalizer makes a different decision on Tuesday than it did on Monday, we can't diff the output and we can't debug a bad take.
The issue author recommends starting with option one.
For long-form English, yes. Start with the rule-based pass and cover the common categories: cardinals, ordinals, years, decimals, currency, percentages, dates, times. Then the symbols, ampersand to and, percent to percent, at-sign to at, hash to number, dollar to dollars, plus to plus, equals to equals. Then context-dependent abbreviations. Then the acronym-versus-initialism split. Then units, and URLs, emails, phone numbers.
The acronym split is the one that's bitten us.
It's bitten everyone. And PolyNorm's prompt has a heuristic for it that I think is the cleanest statement of the rule I've seen. Spell acronyms out to their full forms for clarity, except when the acronym is a widely recognized and pronounceable name, in which case keep it as-is and pronounce it as a word. NASA, NASCAR. And if the acronym combines a letter and a word, split accordingly.
So NASA stays NASA, S-R-ampersand-E-D gets spelled out, and something hybrid gets taken apart.
That's the rule. And it's a rule a human can apply, which is why it works as a prompt.
Now, Daniel asked specifically about small models built for this purpose. Did you find one.
No. And I want to be honest about that, because it's the negative finding of the episode. There is no named standalone small language model built specifically as a TTS text normalizer. I looked. The closest things are PolyNorm, which is LLM-prompt-based but not a small dedicated model, NeMo's neural duplex tagger and decoder, which is a neural TN model but it's part of a larger toolkit, and the vernacula suggestion of just using a generic one-to-three billion parameter instruct model with a rewrite-for-TTS prompt.
So the dedicated small normalizer doesn't exist yet as a product.
The dedicated small normalizer doesn't exist yet as a product. What exists is a set of general tools you point at the problem.
Which is worth saying plainly, because Daniel's framing assumed it existed.
It's a reasonable assumption. It just isn't true yet.
Let me put the asymmetry on the table, because I think it's the sharpest way to state the whole episode.
Go ahead.
ElevenLabs gives you a one-parameter toggle. On, off, auto. Chatterbox gives you a twenty-line punctuation function. That's the gap. The unsafe assumption Daniel flagged isn't really about TTS providers in general. It's about open source specifically. Commercial providers have a business reason to build normalization in, because their customers will notice if the voice reads a phone number as a string of digits. Open-source projects ship the model and leave the preprocessing to you.
And that's not a criticism of Chatterbox. It's a scoping decision. The model is the hard part. Normalization is unglamorous plumbing, and the maintainers reasonably assumed somebody downstream would handle it.
Except nobody downstream handles it, because everybody downstream assumes the model handles it.
Which is exactly the loop the episode is about.
One more thing on the LLM-as-normalizer trend. Is it going to eat rule-based TN entirely.
Not for production pipelines, and I'd bet against it for a while. PolyNorm's numbers are better than the rule-based baseline. But the LLM adds latency, it's non-deterministic, and it costs money per token. For a pipeline that runs once and a human reviews the output, that's fine. For a pipeline that runs thousands of times a day and needs to be reproducible, the rule-based system that gets eighty percent right and never surprises you is often the better engineering choice.
Eighty percent right and predictable beats ninety-five percent right and occasionally creative.
For a lot of use cases, yes. And there's a hybrid that I think is where this lands: rules for the deterministic categories, numbers and dates and currency, and a model for the long tail, the context-dependent abbreviations and the acronym judgment calls.
We've covered the tools, the pairing paradox, and the commercial versus open-source gap. Before we wrap, there's someone here who has actually built one of these by hand.
Hilbert: The binder had a whole page for Doctor.
Sorry, what?
Hilbert: One page. Just for that abbreviation. Because the system would read Doctor Smith as Drive Smith, and it would read Elm Drive as Elm Doctor. Both directions, same page.
You built a normalizer.
Hilbert: I updated one. Summer job, small AM station, automated overnight programming. It was a radio reading service, for blind listeners. The station read newspapers aloud off a text-to-speech system, and the system had a binder of hand-typed pronunciation rules. My job was keeping it current.
Hand-typed.
Hilbert: Hand-typed. We'd get a new one every time somebody heard it get something wrong on air. That binder was a rule-based normalizer, we just didn't call it that. And it capped out around eighty percent, which is the same number your issue report gives.
Eighty percent coverage and then it stops.
Hilbert: It stops. Eventually the station gave up on automating anything with numbers and had a person read the financial pages live. That was cheaper than maintaining the rules.
There's a detail in that binder I want you to confirm, because I don't believe it.
Hilbert: Go on.
You told me once there was a section labeled words we do not say on air. And you said it had nothing to do with pronunciation.
Hilbert: It didn't. It was words the system would read with an emphasis that made them sound sarcastic. Moist was on the list.
Was it actually banned, or are you misremembering.
Hilbert: It's on the list. Page four, about a third of the way down.
Why would a reading service for the blind need to avoid sounding sarcastic about the word moist.
Hilbert: Because it read the weather forecast. Anyway, I've got a thing at four.
A thing at four.
Hilbert: I've got a thing at four.
So the binder was real, the eighty percent cap was real, and apparently moist was banned. Let's bring this back to where we started.
The open question is whether open-source TTS providers eventually ship normalization, or whether it stays a third-party and do-it-yourself problem forever. My guess is it stays DIY, because the model is the interesting part and the maintainers have no incentive to own the plumbing.
And the second open question is whether the LLM approach eats rule-based TN. I think the answer is that latency and non-determinism keep rules alive for production pipelines for a long time yet.
There's a detail from the research that didn't make the main discussion and I think it's worth thirty seconds. The Chatterbox model sizes. Turbo is three hundred fifty million parameters. Nano is one hundred ten million, and it runs three times realtime on eight CPU cores. Multilingual V3 is five hundred million across twenty-three languages.
The smaller those models get, the bigger a share of the total error the normalization layer becomes. Nano at a hundred and ten million parameters is going to be very good at the acoustics and very bad at guessing what a stock ticker is supposed to sound like.
Which means the part of the stack we haven't built may end up being the largest remaining source of errors.
The most model-specific part of the whole thing.
Here's a thought for a future episode, and I'll put it on the table as a pitch rather than a plan. We run on Chatterbox. We've just spent an episode describing the normalization layer we don't have. At some point we could actually build one, the rule-based pass, the acronym dictionary, the internal-caps mitigation, and measure whether it moves the needle on listener complaints. That would be a useful piece of documentation for anyone building on the same model.
I'd listen to that one.
Thanks as always to our producer, Hilbert Flumingtop, who has a thing at four.
This has been My Weird Prompts.
If you want to hear us actually build the thing, or if you've built one yourself, email us at show at my weird prompts dot com. We'll be back soon.
