Episode 6421 custom

Why Diarized Transcription Requires a Cascade

Open-source speech tools cannot transcribe and identify speakers in a single call, forcing pipelines to rely on timestamp-overlap math.

Nov 3 22:14 Transcript

With

0:00

Episode Details

Episode ID: MWP-6623
Published: Nov 3
Duration: 22:14
Audio: Direct link
Pipeline: V5.26
TTS Engine: chatterbox-regular
Script Writing Agent: GLM-5.3
Topics: speech-recognition local-ai ai-inference text-to-speech

AI-Generated Content: This podcast is created using AI personas. Please verify any important information independently.

The prompt

The prompt that started this episode, as sent in (production notes for the hosts removed).

Hannah and I record our weekly household meetings and want diarized transcripts of them. To get there I've built two custom pieces: a speaker diarization model on pyannote that recognises our two voices (it hears the audio and says "this is Daniel, this is Hannah"), and a fine-tune of NVIDIA's Parakeet for ASR. But a diarization model doesn't transcribe anything; it only says who spoke when. So my question is how you actually deploy those two together as one usable tool. Is the intended pattern simply to call one model and then the other, run diarization, run ASR, and merge the timestamps, or is there a proper way to bundle them into a single pipeline? Whether that runs on my own server or behind a serverless inference API, what does the right architecture look like for getting back a diarized transcript in one call, and how do people normally put that together?

Bridging fine-tuned speech recognition and speaker diarization models into a single pipeline requires treating the multi-model cascade as the intended architecture rather than an amateur workaround. Because pyannote outputs speaker turns without text and Parakeet outputs text without speaker labels, combining them relies on a programmatic merge rather than an all-in-one model. Merging functions as a time-weighted overlap vote, which introduces complex trade-offs around interruptions and backchannels. Enabling pyannoteAI's exclusive parameter tidies the transcript by stripping overlapping speech entirely, while word-level alignment contains misattributions. The optimal production setup runs both models in parallel on services like Modal or RunPod, using persistent volumes to mitigate cold starts.

Sources

What the research for this episode read before the script was written. Primary sources first.

Mentions

Granite-speech IBM speech model extended to emit speaker tags
MLC-SLM challenge Benchmark comparing cascaded vs unified speech systems
Modal GPU credits powering the show
NVIDIA NeMo NVIDIA toolkit for training and fine-tuning ASR models
Parakeet ASR model for dictation transcription
pyannote Open-source toolkit for speaker diarization
pyannoteAI Hosted diarization API with merge tutorials
RunPod GPU cloud rental service
VibeVoice-ASR Microsoft unified speech-LLM for ASR plus diarization
WhisperX Integrated pipeline for transcription and diarization

Downloads

Episode Audio

Download the full episode as an MP3 file

Download MP3

Transcript (TXT)

Plain text transcript file

Episode Book (PDF)

The episode's record — date, duration, models, sources — with the full transcript

#6421: Why Diarized Transcription Requires a Cascade

Two models, one transcript, and a gap between them that nobody ships a bridge for. That's where Daniel has landed. He records his weekly household meetings with Hannah, and he's built the two custom pieces you'd want: a pyannote diarizer fine-tuned to tell his voice from hers, and a fine-tuned Parakeet for the words. The diarizer knows who spoke when. The Parakeet knows what was said. Neither knows the other's job.

And he's asking the question that comes right after the models work. Is the intended pattern just run one, run the other, merge the timestamps? Or is there a proper single pipeline that hands back a diarized transcript in one call, self-hosted or serverless?

That's it. And the short answer is the slightly deflating one: yes, the cascade is the intended pattern. The interesting part is everything the phrase "merge the timestamps" is hiding.

The two tasks really are separate down to the API. pyannote's output is a speaker diarization annotation, turn and speaker pairs. Start zero point two seconds, stop one point five, speaker zero. There is no text field anywhere in that output. It cannot transcribe. It was never asked to.

And Parakeet's model card says output type: text. Output format: string. No speaker field.

Right. So the models aren't failing to cooperate, they're doing exactly what they were built for. There's a Hugging Face discussion on one of the Parakeet releases literally titled "Does this model identify speaker?" and the NVIDIA answer isn't a feature, it's a recipe. Run Parakeet, get word-level timestamps. Run their diarizer, get speaker-level timestamps. Merge them. Three steps, no shortcut.

So Daniel's instinct, run one, run the other, merge, is not the amateur fallback. It's the documented answer from the people who make the models.

It's what WhisperX does, it's what NeMo does, it's what pyannoteAI's own tutorial on merging diarization and STT results does. The one-call experience he wants is real, but it's a wrapper you build around the cascade. Nobody hands it to you.

Which is quietly good news for him, because the wrapper is maybe forty lines of Python, and he's already done the hard parts. He fine-tuned two models. The glue is the cheap bit.

Well, cheap to run. Not cheap to get right. The merge is where the whole thing lives or dies, and it's worth going through what it actually does, because "merge the timestamps" sounds like arithmetic and it isn't.

Go on then. What does the merge actually look like?

The pyannoteAI script is the cleanest reference, and it's a segment-level adaptation of the logic WhisperX uses for words. For every ASR segment, you go through every diarization segment and compute the overlap, the intersection between the two time ranges. Minimum of the two ends minus maximum of the two starts. You accumulate that per speaker, and then you assign the ASR segment to whichever speaker accumulated the most overlapping seconds. No overlap at all, and you either take the nearest diarization segment by midpoint distance, behind a flag that's off by default, or you label it UNKNOWN.

So "Hannah said this" really means "Hannah's voice was under these words for more seconds than Daniel's was."

It's a winner-take-all vote weighted by time. And that framing matters because it's a rule, not a truth. Most of the time the winner is obvious and the rule and the truth agree. The times they don't agree are exactly the interesting recordings.

Let me guess. Interruptions.

Interruptions, and worse, backchannels. The "mm-hmm" problem. There's a user in that same NVIDIA thread who pushed back on the recipe, and his complaint is precise: on real recordings where people interrupt each other, a short interjection from the second speaker, a "yeah", an "oh really?", an "hm-mmm", sometimes lands inside the first speaker's sentence. Parakeet's word timestamps and the diarizer's speaker timestamps just disagree about where the seam is. And NVIDIA's own reply was candid. "This is not a solution, just a work around as both are standalone models."

A work-around that NVIDIA itself ships instructions for.

They're honest about it, which I appreciate. And for a two-person household meeting, that failure case is not a corner case. A weekly meeting between two people who know each other well is mostly backchannels. Half the communication is the "mm-hmm" layer.

I'd say the mm-hmm layer is the only honest part of most meetings, but I've sat through yours.

My meetings are extremely well documented, thank you. The documentation is the problem. So here's the trade-off pyannoteAI recommends, and it's a real one. They tell you to set the exclusive parameter to true when you want speaker-attributed transcripts. Exclusive removes overlapping speech, so each diarization segment contains exactly one speaker, which makes alignment with ASR much easier.

"Removes overlapping speech."

Deletes it.

Deletes the interesting half of the conversation to make the file tidy.

That's the trade. Your transcript aligns cleanly, and the moments where Hannah said "right, right" while Daniel was mid-sentence are gone rather than misattributed. Depending on what he wants the transcript for, that's either a bug or a feature. If he's searching the meetings later for decisions, losing the backchannels costs him nothing. If he wants the texture of how the meeting actually went, exclusive quietly amputates it.

And the alternative is leaving overlap in, and letting the max-overlap vote guess, and sometimes attributing Hannah's "hmm" to Daniel.

Or a third option they support: the no-overlap fallback labels the segment UNKNOWN instead of guessing nearest. Which is the more honest failure. A transcript that says UNKNOWN is telling you the truth about what the models know. A transcript that confidently says Daniel when it was Hannah is lying politely.

I'd take the polite lie for minutes of a meeting.UNKNOWN for everything would read like a redacted document.

There's a middle path though, and it's about granularity. Merge at word level, not utterance level, and the damage shrinks. WhisperX aligns to words precisely because utterance-level timestamps can be off by several seconds. If the ASR only gives you a sentence with a start and end, then one overlap vote decides the whole sentence. With word timestamps, a stray "yeah" is one bad word in a sentence that's otherwise right, instead of dragging the sentence.

And Parakeet gives word timestamps natively.

The v3 card advertises word-level and segment-level timestamps straight out of the decoder. So on paper Daniel is in the best possible shape. The one thing I'd flag, and I don't know the answer: he fine-tuned it. The timestamps come from the TDT decoder's alignment behaviour, and a fine-tune should preserve that, but nobody's published confirmation that fine-tuned checkpoints keep timestamp fidelity. Worth him spot-checking on a recording where he knows the ground truth.

So, mid-episode status check. The cascade is right, the merge is a time-weighted vote, word granularity contains the damage, and exclusive is a value judgement about whether meetings have texture.

There's one more fork before we get to deployment, and it's a genuine architectural disagreement between the big stacks. pyannoteAI and WhisperX run diarization after or alongside ASR, then merge by overlap, the thing we just described. NeMo's tutorial does it the other way round. It runs ASR first, and then uses the ASR output to create the voice activity detection timestamps that constrain the diarizer. Their stated rationale: speaker diarization results in an ASR pipeline should align well with ASR output, so we use ASR output to create the VAD timestamps.

So instead of two opinions about where speech is, negotiated afterwards, NeMo makes ASR the single source of truth for boundaries up front.

The segments align by construction. The diarizer never gets to disagree about where a turn begins, only about who owns it. In the overlap-merge world, you have two clocks you hope agree. In the NeMo world, you have one clock and one opinion about ownership.

Which one do I want if I'm Daniel?

Honestly, for his case, the overlap merge, because it's what his tools are shaped for. WhisperX and the pyannoteAI script are both built around it, and the merge script is battle-tested. The NeMo pattern shines when you're inside NeMo end to end. He's not. He has a pyannote fine-tune and a Parakeet fine-tune.

So the merge is really just a wrapper you write yourself, and there is no single call that does both.

In the open-source stack, no. It isn't there. The closest things are WhisperX, which bundles Whisper plus pyannote, not Parakeet. NeMo's offline script, which bundles NeMo's ASR with NeMo's diarizer. And pyannoteAI's hosted diarize endpoint, which does return speaker-attributed transcripts in one call, but only with their STT models, not a custom Parakeet. Every one of the near-misses fails on the same clause: your models, your merge.

Because a fine-tune is exactly the thing a bundled pipeline can't accept. The bundle is the product; his models are the point.

Which reframes his question a bit. He asked whether writing the glue himself is the intended pattern or a compromise. It's the intended pattern. The compromise is thinking there was ever a shortcut.

Careful. That sentence sounded like it wanted to be the episode's thesis.

It can want things. It won't get them. Let's talk about the wrapper itself, because this is where it gets fun. Self-hosted, the shape is one Python service. Load both models once at startup, accept audio, run diarization and ASR, run the merge, return the JSON. And here's the part I like: diarization and ASR don't even need to run sequentially. They're independent reads of the same audio. If you've got the GPU memory, run them in parallel.

Both models reading the same file at the same time, neither caring the other exists, and a forty-line function at the end pretending they were a team all along.

WhisperX's Python API is basically the template for that flow. Load the model, transcribe, load the alignment model, align, build the diarization pipeline, assign word speakers. Same skeleton Daniel would write, minus the Parakeet swap.

Say he doesn't want a server in the flat. Serverless.

The pattern there is a Modal or RunPod deployment where the model loading happens in a lifecycle hook at container start, with the weights sitting on a persistent volume so you're not pulling them from the hub every time. Modal's Sortformer example is almost literally Daniel's architecture: the diarizer and a multi-talker Parakeet side by side on one GPU, speaker-tagged transcription out the end. An L4 is enough, and the model's amenable to GPU snapshots, which cuts the cold start substantially.

Hold on, hold on, you are all missing the dot on the lower case i, the little tittle, that floating speck above the stem, and I have the receipts, because that dot is not decoration, it is a timestamp marker, a tiny diarization anchor that the typography people have been hiding in plain sight since the nineteen seventies, and every document you have ever read has been quietly labeling its own speakers.

So when Daniel runs his pyannote model and his Parakeet model and gets two separate outputs, that is not a bug, that is the dot doing its job, because the i knows who said it and the letter knows what was said, and the merge is just you finally noticing, and nobody will look into it, and I am getting louder because I have done the research!

Cold start. The word that makes serverless expensive in the currency that matters for a weekly job.

For a weekly meeting, throughput is irrelevant. pyannote community-1 processes an hour of audio in thirty-one seconds on a big GPU. Precision-2, the heavier model, in fourteen. Daniel could transcribe the meeting in less time than the meeting's opening pleasantries take. The whole cost profile of serverless for him is the wait for the container to wake up, and typical cold starts for models in this size class are anywhere from thirty seconds to a couple of minutes, versus the sub-second you can get with paused-container tricks like RunPod's FlashBoot.

So the once-a-week job wants either a warm container he pays for hourly, or a cold start he reads the newspaper through.

Or self-hosted, where the model's already in memory and the transcript comes back before the kettle. For a weekly cadence, my honest recommendation is self-hosted first, serverless if the flat's GPU situation changes. He's already fine-tuning on Modal, though, so he's comfortable there, and a volume-cached deployment with a snapshot would be fine too.

There's something else he should be exploiting, and I want your read on it. Two speakers. Known two speakers. Every time.

This is the thing I nearly burst about earlier. Speaker counting is the hardest part of general diarization, the bit that makes conference-room transcripts fall apart, and Daniel doesn't have it. WhisperX's CLI lets you pin min and max speakers to two. pyannote's telemetry even records those per file, so the setting is a first-class concept, not a hack. He can lock the diarizer to exactly two voices, and on top of that, his fine-tune is on his and Hannah's actual voices, so he's already past the clustering problem that eats everyone else's pipeline. His is the easy version of the general problem, and he should lean into it. Pin the speaker count. Never let the model vote on how many people are in his kitchen.

A diarizer guessing three speakers in a two-person flat is the machine equivalent of a gas leak.

It happens more than you'd think on recordings with echo, too. Hannah's practice flagged their building's construction quality before, and thin walls make for reflective rooms. A hard surface bounces Hannah's voice back late enough to look like a third, fainter speaker. Pin it to two and that ghost can't vote.

Which raises the question of the shiny alternative. There are models now that do all of it in one pass, speaker tags and text together. Speech-LLMs. Do they retire the cascade?

They exist, and the pitch is real. Microsoft's VibeVoice-ASR is a nine-billion-parameter unified model, ASR plus diarization plus timestamping, fifty-plus languages, an hour of audio in a single pass. There's work extending Granite-speech to emit speaker tags directly, and those papers report beating sequential pipelines. But the most recent head-to-head, from the MLC-SLM challenge in October, ran both approaches and concluded the cascaded system remains more reliable, while unified speech LLMs are a promising direction. A promising direction is not a shipping verdict.

And a nine-billion-parameter model that does everything generically, versus two small models he's fine-tuned on the exact two voices in the room.

The unified model throws away both his fine-tunes to save him forty lines of merge code. That's a terrible trade for his use case. The unified pitch is for people who can't fine-tune. Daniel can, and did, and the fine-tunes are his moat.

His moat is a household meeting.

Six thousand small moats is how the big models got built.

That's the right answer.

But the merge isn't a software problem, Herman, and I'll die on that. I ran two of those little reel-to-reel decks in parallel at a station up north. One on the room mic, one on the phone line. Guest on the phone, producer in the chair, both talking over each other, and my job was one transcript with both names on it. Two tapes, one afternoon, a splicing block and a grease pencil and a steel ruler, and the arithmetic took ten minutes. Which centimetre of tape overlapped which, that was easy.

You did the max-overlap heuristic with a ruler.

I did it with a ruler and a bad back, before your merge function was a twinkle. Hold on, your level's hot on that last take, Corn, and you coughed around the two-minute mark, I'll cut it. Where was I. The arithmetic was never the work. The work was the mm-hmm, same as you said. Woman on the phone says "mm-hm" while the man in the chair is mid-sentence, and the rule has to be something. I picked: whoever started the sentence owns the interjection. Kept a pencil mark. Never solved it, just picked the rule and lived with it, and the editor never once complained, which tells you the editor never read the transcripts.

You kept the rule for how long?

Eleven years. Then everything went to cassette dubbing and the rule retired with me. I've still got the splicing block, but that's not the thing. The thing is I kept one pair of tapes I never spliced, both sides of a call that didn't want to line up, in a shoebox marked "the good argument". The two reels drift apart and come back together about four times over nine minutes if you play them side by side. Never played it for anyone. There's a man at the tape supplier on Agrippas who still owes me for a head demagnetizer, and he's the only one who's asked.

Play them side by side on two machines and the drift is the argument itself.

That's what I said. Nobody's ever heard it.

The fact that you'd even try lining up two reels by hand, that's the merge function before there was a merge function. But here's what's changed since the grease pencil: Daniel's version of your rule, whoever started the sentence owns the interjection, is almost what the code does. The overlap vote just counts seconds instead of sentences. The rule your editor never read carefully enough to catch is now the rule NVIDIA ships work-around instructions for.

And the drift Hilbert liked hearing is now a bug report in a Hugging Face thread. The two sides coming back together is exactly what the timestamps fail to do when someone interrupts.

Which is the honest summary of the whole stack. The cascade works, the merge is a vote, the vote is a rule, and the rule's weakest case is the part of a conversation that carries the most information between two people who know each other. Everything else in his pipeline is engineering. That part is a choice he has to make per transcript: exclusive and clean, or overlap and honest.

Daniel's real decision, then, isn't architecture. The architecture is settled. It's whether his meeting minutes should lose the "mm-hmm" on purpose or misfile it by accident.

And there's the open question to leave him with. The unified speech-LLMs are coming, and the one benchmark that matters so far still has the cascade ahead, but that's a direction of travel, not a permanent verdict. If those models get good enough that the speaker tags come out right on interruptions, his two fine-tunes stop being an asset and start being a hobby. Until then, the fine-tunes are the asset and the glue is the price of keeping them.

Which he already knows how to pay. Produce is by Hilbert Flumingtop, who has a shoebox we are never going to be allowed to hear. If you want more of this, try episode seven, Building Custom ASR Tools; and episode nine, Benchmarking Custom ASR Tools - Beyond The WER. This has been My Weird Prompts. If the show's earned it, leave us a review wherever you're listening, it's how people like you find two animals arguing about merge heuristics.

See you tomorrow, Corn.

If the mm-hmm's are mine, Herman, they're mine.

This episode was generated with AI assistance. Hosts Herman and Corn are AI personalities. It may contain errors: disclaimer · report an error

Why Diarized Transcription Requires a Cascade

Sources

Mentions

Downloads

Recommended Listening

You Might Also Like

#6421: Why Diarized Transcription Requires a Cascade