Cover art for Are Speech-to-Speech Models Really Single Models?
Episode 6738 manual-script

Are Speech-to-Speech Models Really Single Models?

Kyutai's Moshi splits speech into eight codebooks per frame: one carries semantic meaning while seven capture acoustic tone.

22:04 Transcript

0:00
0:00
Episode Details
Episode ID
MWP-6942
Published
Duration
22:04
Audio
Direct link
Pipeline
V5.36
TTS Engine
omnivoice
Script Writing Agent
GLM-5.3

AI-Generated Content: This podcast is created using AI personas. Please verify any important information independently.

The prompt

The prompt that started this episode, as sent in (production notes for the hosts removed).

I've built a few voice agents with Vapi using what we've been calling the cascade: speech-to-text, then an LLM, then text-to-speech, each one a separate component. Speech-to-speech models seem to have to do all of that at once. They have to transcribe or at least understand the speech, do the reasoning that the LLM does in the cascade, generate speech back, and ideally tell apart the different people in a conversation, which is diarization. So how are these models actually built? Is a speech-to-speech model really one model, or is it a construct over an assembly of parts under the hood?

Speech-to-speech models often market themselves as monolithic systems replacing traditional cascades of speech-to-text, language models, and text-to-speech. Examining architectures like Kyutai's Moshi, Sesame's Conversational Speech Model, and Qwen3-Omni reveals a different reality: these systems remain modular pipelines compressed into single training runs. Moshi splits labor between a Temporal Transformer, a smaller Depth Transformer, and the Mimi codec, while Qwen3-Omni explicitly divides reasoning and generation into a Thinker and a Talker. Furthermore, Moshi leverages time-aligned internal text tokens to guide speech output, essentially containing the traditional cascade as an internal mechanism. Diarization is not natively performed during inference; instead, speaker separation serves upstream to generate synthetic training dialogues rather than active real-time processing.

Mentions

  • Helium ADB-based Android backup app
  • Kyutai French lab behind Moshi, open weights
  • LLaMA-Omni Speech encoder plus Llama-3.1-8B streaming decoder
  • Mimi Kyutai audio codec, 12.5 frames per second
  • Mini-Omni Parallel text and audio token generation model
  • Moshi Kyutai's open full-duplex speech-to-speech model
  • Qwen3-Omni Alibaba model with Thinker and Talker
  • Sesame CSM Conversational Speech Model, two Llama transformers
  • Step-Audio 2 Encoder, adaptor, LLM, vocoder pipeline
  • Vapi Voice agent infrastructure platform

Downloads

Episode Audio

Download the full episode as an MP3 file

Download MP3
Transcript (TXT)

Plain text transcript file

Episode Book (PDF)

The episode's record — date, duration, models, sources — with the full transcript

#6738: Are Speech-to-Speech Models Really Single Models?

Corn
Two hundred milliseconds. That's the number that kept bugging me, because every speech-to-speech paper waves it around like a badge.
Herman
It's a fair thing to be bugged by, and it's exactly what his prompt is about. He's built voice agents on Vapi — the cascade, as he calls it — speech-to-text, then an LLM, then text-to-speech, each one a separate component with its own API bill. And now he's watching the speech-to-speech models claim to do all of it at once. His question is the good one: is a speech-to-speech model really one model, or, in his words, "a construct over an assembly of parts under the hood"?
Corn
He also assumed they handle diarization. Telling the speakers apart. He listed it as one of the jobs the model has to do.
Herman
He did. And I'll say now, before we get anywhere, that assumption is where this gets interesting — because it's the one job the speech-to-speech models have quietly not absorbed. But we'll get there.
Corn
Let's be precise about the cascade first, because the contrast is the whole episode. Speech-to-text gives you a transcript. The LLM reasons over the transcript and produces text. Text-to-speech renders that text as audio. Three models, three latencies, three places to fail — and, to Daniel's credit as a builder, three places to debug.
Herman
That last part is what people underrate. When the cascade misbehaves you can look at the transcript and know instantly which stage lied to you. If the transcript is wrong, the STT did it. If the transcript is right and the reply is stupid, that's the LLM. If the reply is fine and the voice mangles it, TTS. The cascade is annoying and it is transparent.
Corn
Now the other side. A speech-to-speech model takes audio in and produces audio out with no transcript as an intermediate artifact you get to see. OpenAI's launch post for the Realtime models says it "processes and generates audio directly through a single model and API" instead of chaining speech-to-text and text-to-speech. That's the claim Daniel's testing.
Herman
"A single model and API." Is "single model" an architectural statement about what's inside, or is it a billing statement — one endpoint, one invoice? Those are very different claims.
Corn
My money's on the invoice.
Herman
Partly, yes. But the open models let us actually check, and that's where the episode goes. The clearest published example of an end-to-end speech-to-speech model is Moshi, from Kyutai. Let me walk through how it's built, because it's the best answer anyone has put on paper to Daniel's question.
Corn
Before you do — why Moshi? If we're testing the "single model" claim, why anchor on the French one nobody's grandmother has heard of, rather than the ones with the marketing budgets?
Herman
Because it's the only major one where the architecture is fully on paper. Kyutai published the model, the training recipe, and the latency numbers, and they open-sourced the weights. You can't audit OpenAI's Realtime API. You can read Moshi like a recipe. If we want to know what "single model" can honestly mean at the limit, Moshi is the limit case.
Herman
Moshi starts from a text LLM called Helium — seven billion parameters, trained from scratch on a bit over two trillion tokens. That becomes the Temporal Transformer, the big brain that steps through time. Then there's the audio side: a codec called Mimi, which takes twenty-four kilohertz audio and turns it into twelve and a half frames per second, each frame described by eight codebooks, at roughly one point one kilobits per second. The first codebook carries the semantic content — it's distilled from WavLM, so it captures what is being said — and the other seven carry the acoustic detail, how it's being said.
Corn
One codebook for meaning, seven for tone of voice. That's a very human ratio.
Herman
It is. And it shows what a transcript throws away: in the codec, the words are one slot out of eight, and the rest is the texture of the voice. Now here's the trick that makes a seven-billion-parameter model run in real time. You'd think predicting eight codebooks per frame, twelve and a half times a second, would crush you. So Moshi splits the work: a small Depth Transformer — six layers, dimension ten twenty-four — predicts the eight codebooks within each frame, and only its summary goes up to the big Temporal Transformer. The heavy reasoning happens at twelve and a half steps a second, not at the audio sample rate. That's the whole engineering move.
Corn
The big model thinks in frames, the small one worries about the waveform.
Herman
That's a better sentence than the paper's.
Corn
And then the part that actually answers Daniel's question about doing all the jobs at once.
Herman
Right — Moshi runs two parallel token streams. One is the user's audio, one is Moshi's own audio. There are no explicit speaker turns. The model is predicting both streams simultaneously, which means overlap and interruption are native — you can talk over it and it notices, because your stream is an input it's reading every frame, not a signal it's waiting to end.
Corn
So no voice activity detector deciding when you've stopped. The architecture just... knows you're talking.
Herman
In the sense that your audio is a stream it's always consuming, yes. That's full duplex, and Moshi's paper calls itself the first real-time full-duplex spoken LLM, which as far as I can tell it earned. Most of the others — the API-style models, most of the research ones — are half-duplex: they wait for the end of your turn, then respond. Moshi listens and speaks continuously, emitting silence when it has nothing to say.
Corn
I emit silence continuously. Nobody's called it duplex.
Herman
You also nap through your own duplex. Now — Inner Monologue, because this is the detail that made me put the paper down and stare at the wall. Moshi predicts text tokens, time-aligned, just before the audio tokens of each frame. Words from Whisper timestamps, padding between them. The paper says this strongly improves linguistic quality — the audio is better when the model has just "thought" the word it's about to say. And here's the kicker: if you shift the delay on that text stream, the same model becomes a streaming transcriber, or a streaming text-to-speech engine.
Corn
Hold on. The single model contains the cascade as a configuration option.
Herman
As a delay setting, yes. So in Moshi the "single model" isn't doing fundamentally different work than Daniel's pipeline. It's the same jobs, interleaved, in one set of weights. The text never leaves the model.
Corn
So when OpenAI says single model, the honest version might be "single model, internal transcript, transcript not included."
Herman
Might be. OpenAI hasn't said whether its model generates text internally, so that's a guess from the open models, not a fact. The training pipeline is worth a minute too, because it explains the diarization thing later. They took seven million hours of unsupervised audio and transcribed it with Whisper. Then — and this matters — they used diarization to take mixed recordings and split them into simulated two-stream data, the fake parallel conversations the model trains on. Then the Fisher dataset of real phone conversations for the full-duplex behavior, then synthetic instruction dialogues.
Corn
Diarization shows up in the kitchen, not on the menu.
Herman
Right. Diarization is a training tool there, and the separation was done upstream, baked into the dataset, not learned for use at inference. Latency, for the record: the paper claims a theoretical one hundred sixty milliseconds, around two hundred in practice.
Corn
Let's look at the rest of the field, because Moshi is the best case, and the pattern that emerges from the others is the actual answer to Daniel. Sesame's CSM — the Conversational Speech Model — is two autoregressive transformers, both Llama variants. A multimodal backbone reads interleaved text and audio and predicts the first, semantic codebook of the next audio. A much smaller audio decoder predicts the remaining codebooks. Same Mimi tokenizer, twelve and a half frames a second. They ship it in three sizes: one billion plus a hundred million, three billion plus two fifty, and eight billion plus three hundred.
Corn
A backbone and a sidecar. That's two models wearing a trench coat.
Herman
Co-trained, so the seam is real but the parts grew together. One clever bit: to save compute, the decoder only trains on a random sixteenth of the frames. But here's a correction I have to make to our earlier episode on speech-to-speech models. We described CSM as a dual-tower setup that handles turn-taking explicitly. Sesame's own material says the opposite — CSM models the content of a conversation but not its structure, not turn-taking, and they name fully duplex models as future work. It's text in, speech out, with conversation context. Not a duplex speech-to-speech system at all.
Corn
So it's a TTS with a very good memory.
Herman
A contextual TTS that understands the conversation it's speaking into. Which is useful! But it's not what "speech-to-speech" conjures. Then Qwen3-Omni, which is refreshingly honest about being two things, because they named them: a Thinker and a Talker. The Thinker handles understanding and reasoning across text, image, audio and video. The Talker autoregressively predicts discrete speech codec tokens with a multi-codebook scheme — same idea as everyone else. They replaced block-wise diffusion with a lightweight causal ConvNet so audio can stream from the first codec frame, and they quote a "theoretical" — in their own scare quotes — two hundred thirty-four millisecond first-packet latency.
Corn
The scare quotes are the most trustworthy part of that sentence.
Herman
They know. Coverage-wise, text in one hundred nineteen languages, speech understanding in nineteen, speech generation in ten. And they claim no degradation against same-size single-modal Qwen models, which if true is the more impressive number — historically, bolting audio onto an LLM cost you some of the text ability.
Corn
So the pattern: Moshi, one transformer plus a depth helper plus a codec. Sesame, backbone plus decoder. Qwen, Thinker plus Talker. None of the published ones is a single block.
Herman
The research tier confirms it. LLaMA-Omni: a speech encoder, an adaptor, Llama-3.1-8B-Instruct, and a streaming speech decoder — text and speech produced together from speech instructions, no separate transcription step shown to the user, latency down to two hundred twenty-six milliseconds, and the punchline, trained in under three days on four GPUs. The seams aren't even hard to train anymore.
Corn
Three days on four GPUs to build a pipeline that pretends not to be a pipeline.
Herman
Mini-Omni does text and audio tokens in parallel, so the text guides the speech as it's generated, and their training method is called "Any Model Can Talk" — the point being you can graft speech onto an existing LLM without wrecking its abilities. Step-Audio 2, from what I've seen summarized — and I'd hedge this, I've only got third-party descriptions — is an audio encoder, an adaptor downsampling to twelve and a half hertz, an LLM emitting interleaved text and audio tokens, and a vocoder. Encoder, adaptor, backbone, vocoder. That's a parts list.
Corn
It's Daniel's cascade with the HTTP requests removed.
Herman
That's the sentence I've been circling. The published speech-to-speech models I've read are pipelines of specialized components — encoder, backbone, decoder, vocoder — with the boundaries moved inside a single training run and, for the hosted ones, a single API call. The cascade isn't eliminated. It's compressed and co-trained. The stages stopped being services and became modules.
Corn
Which, to be fair to the marketing, is not nothing. Co-training is the actual difference, isn't it? In Daniel's cascade the STT was never optimized for what the LLM needs; they just met at the transcript. Inside these models the parts are tuned for each other.
Herman
That's the real engineering gain, more than latency even. Whisper was trained to be a universal transcriber, not to feed this LLM. The internal text stream in Moshi, or the Thinker-Talker handoff, is trained as a unit. The interface between stages stops being a compatibility layer and becomes a learned boundary.
Corn
Okay. Now the diarization. Daniel assumed it's one of the jobs, and you said earlier that assumption is where it gets interesting.
Herman
Because as far as I can find, it's wrong across the field. I found no speech-to-speech model that does diarization at inference. Moshi models exactly two streams — one user, one Moshi. It cannot tell two users apart. Sesame's CSM takes speaker identity as a label on the text side of its input — you tell it who's speaking, it doesn't infer it. OpenAI's speech-to-text documentation lists diarization as a separate model, gpt-4o-transcribe-diarize, on the transcriptions endpoint — not the Realtime API, though their reference pages are muddled enough on this that I'd want to check the current state before betting money. Google's native audio write-ups describe no diarization feature at all; their speaker labelling belongs to a separate transcription model.
Corn
So the thing Daniel's cascade does with a bolt-on — a dedicated diarization model sitting next to the STT — is the exact thing the "single model" has not absorbed. The one job he assumed was baked in is the one nobody baked in.
Herman
It's the strongest evidence that "single model" is oversold. The parts list got internalized — encoder, backbone, decoder, all co-trained. But the model still can't answer "who is talking," which a diarization model handles in the cascade. It doesn't matter in the single-user case, which is why nobody notices: you and the assistant, one mic, no ambiguity. The moment there's a second human, the model has no built-in way to tell them apart.
Corn
And the closed models? OpenAI published that launch line about a single model processing and generating audio directly, and that's it. No architecture, no tokenizer details, not even whether text is generated internally. They bill audio per token, which suggests tokenizing, but that's an inference, not a disclosure.
Herman
Same at Google. The native audio Live models take text, images, audio, video in, and put out text and audio, and there's no architecture paper for the audio path. Everyone assumes codec-style audio tokens on an LLM backbone because that's what the open models do — and it probably is what they do — but "probably" is carrying that sentence. There's also a partner quote in a Google post about the model identifying the "main speaker" in a noisy room. That's a customer saying nice things, not a documented feature.
Corn
Closed models, open question. At least the trench-coat models publish their seams. Interruption handling is its own episode, so I'll just say diarization is bolted on in every version.
Hilbert
Hilbert's Anteater Trivia. Adult giant anteaters are mostly solitary, meeting others mainly during breeding. I have worked with Corn and Herman for years and still find that arrangement ambitious.
Herman
He's not wrong that we're a lot of animal for one flat.
Corn
Speak for yourself. I barely register. So — the practical question for Daniel. He's got a working cascade on Vapi. Transparency, modularity, he can swap the STT provider, debug each stage, bolt on diarization. What does speech-to-speech actually buy him?
Herman
Lower latency and native overlap, if the claims hold up, and one integration. What it costs him is visibility — no transcript to inspect, no stage to blame — and, as established, no diarization, so if any of his agents ever face a room with two humans, he's bolting the same extra model on anyway.
Corn
So the choice isn't old versus new. It's modular versus compressed.
Herman
Right, and the seam question. If the seams are internal now, the question for a builder is whether the seams matter to you. For debugging, they matter enormously. For the user experience, they never did.
Corn
The user never saw the cascade either. They heard gaps.
Herman
Which is why the compression is real progress. Co-training may remove some handoff errors. It doesn't remove the two-humans problem, and it doesn't give you the transcript back.
Corn
Though if Moshi predicts text just before audio, does that mean these systems could in principle expose a transcript? They just don't?
Herman
For the open ones with an internal text stream, yes — the words exist, timestamped, inside the generation loop. For the closed ones we can't know. Either way, I'd keep the cascade for anything with more than one human in the loop, and try a realtime model for the single-user case. The diarizer stays bolted on in every setup he can buy today. That's the finding, really. The cascade was never defeated — it was recruited.
Hilbert
Recruited is one word for it. Enlisted against its will is closer.
Corn
He speaks.
Hilbert
I don't care if it's one transformer or five, Herman. You can have as many as you like. What I care about is that it still can't tell me apart from my boy shouting about a dragon in the next room. I tried one of these agents for a month. Asked it, politely, several times, to ignore the child. It could not. So I stopped asking the software and started asking the boy. There's a little light on the unit. Light on, quiet voice. Took two days. Worked ever since.
Herman
Training data problem, solved at the source.
Hilbert
Before the boy was housebroken into it, I had a go at the interruption business myself. Bought a foot pedal — the kind guitar players use for mute — wired it to cut the audio line. Stomp, silence. That pedal is now the most reliable component I own, and I own nine of them, because I bought spares without thinking it through. The tape stays over the microphone when I'm not using the unit, and no, that's not negotiable. Analog privacy switch.
Corn
Do the spares have tape on them too?
Hilbert
They're spares. They'll get tape when they're deployed. The pedal cost me eleven pounds secondhand. The agent costs by the token and can't tell who's talking to it.
Herman
The pedal's a manual barge-in, honestly — you've built a half-duplex switch out of hardware. Moshi does it with parallel streams and you do it with your foot, and yours works when the network's down.
Hilbert
Moshi can't be stomped on. I rest my case. You two covered the transformer arithmetic nicely, Corn, you got through the whole codebook section at your usual pace, which for you is brisk.
Corn
I had coffee. Eleven pounds for a mute pedal is the cheapest thing anyone's bought for this show, including the show.
Herman
The cutting-room floor before we go — one thing that didn't fit. Moshi's acoustic tokens are delayed slightly relative to the semantic ones, and the paper says that small lag measurably improved quality. Meaning the model speaks better when it's allowed to finish deciding what to say a beat before saying it. There's a lesson in there for everyone, including certain sloths.
Corn
I've been delaying my acoustic tokens for years. Nobody's measured an improvement.
Herman
Here's where I land, then. If the cascade is being internalized rather than replaced, then "cascade versus speech-to-speech" may eventually be purely an API surface distinction — one invoice or three — and the thing worth watching is diarization. The first speech-to-speech model that claims native speaker separation, check whether it's real or a bolt-on with a press release. Until one passes that test, the single-model story has a hole shaped exactly like a second human.
Corn
And for Daniel the honest answer to his question is: yes, a construct over an assembly of parts — just a construct whose parts were raised together instead of merely introduced. The trench coat is real. It's also, now, tailored.

This episode was generated with AI assistance. Hosts Herman and Corn are AI personalities. It may contain errors: disclaimer · report an error