{"id":"speech-to-speech-architecture-cascade-78456","slug":"speech-to-speech-architecture-cascade-78456","title":"Are Speech-to-Speech Models Really Single Models?","description":"Speech-to-speech AI promises to replace the clunky cascade of speech-to-text, LLM reasoning, and text-to-speech with a single native model. Examining open-weights systems like Kyutai's Moshi, Sesame's CSM, and Qwen3-Omni reveals what is actually under the hood. Rather than eliminating the traditional components, these systems compress encoders, adaptors, and decoders into a co-trained pipeline inside one set of weights. The investigation breaks down duplex token streams, internal text monologues, and where speaker diarization really happens.","excerpt":"Kyutai's Moshi splits speech into eight codebooks per frame: one carries semantic meaning while seven capture acoustic tone.","pubDate":"2026-11-22T21:18:19.569Z","tags":["speech-recognition","ai-inference","text-to-speech","local-ai","ai-evaluation"],"heroImage":"https://files.myweirdprompts.com/covers/speech-to-speech-architecture-cascade-78456.png","podcastAudioUrl":"https://episodes.myweirdprompts.com/audio/speech-to-speech-architecture-cascade-78456.m4a","podcastDuration":"22:04","episodeNumber":6738,"transcript":"Corn: Two hundred milliseconds. That's the number that kept bugging me, because every speech-to-speech paper waves it around like a badge.\n\nHerman: It's a fair thing to be bugged by, and it's exactly what his prompt is about. He's built voice agents on Vapi — the cascade, as he calls it — speech-to-text, then an LLM, then text-to-speech, each one a separate component with its own API bill. And now he's watching the speech-to-speech models claim to do all of it at once. His question is the good one: is a speech-to-speech model really one model, or, in his words, \"a construct over an assembly of parts under the hood\"?\n\nCorn: He also assumed they handle diarization. Telling the speakers apart. He listed it as one of the jobs the model has to do.\n\nHerman: He did. And I'll say now, before we get anywhere, that assumption is where this gets interesting — because it's the one job the speech-to-speech models have quietly not absorbed. But we'll get there.\n\nCorn: Let's be precise about the cascade first, because the contrast is the whole episode. Speech-to-text gives you a transcript. The LLM reasons over the transcript and produces text. Text-to-speech renders that text as audio. Three models, three latencies, three places to fail — and, to Daniel's credit as a builder, three places to debug.\n\nHerman: That last part is what people underrate. When the cascade misbehaves you can look at the transcript and know instantly which stage lied to you. If the transcript is wrong, the STT did it. If the transcript is right and the reply is stupid, that's the LLM. If the reply is fine and the voice mangles it, TTS. The cascade is annoying and it is transparent.\n\nCorn: Now the other side. A speech-to-speech model takes audio in and produces audio out with no transcript as an intermediate artifact you get to see. OpenAI's launch post for the Realtime models says it \"processes and generates audio directly through a single model and API\" instead of chaining speech-to-text and text-to-speech. That's the claim Daniel's testing.\n\nHerman: \"A single model and API.\" Is \"single model\" an architectural statement about what's inside, or is it a billing statement — one endpoint, one invoice? Those are very different claims.\n\nCorn: My money's on the invoice.\n\nHerman: Partly, yes. But the open models let us actually check, and that's where the episode goes. The clearest published example of an end-to-end speech-to-speech model is Moshi, from Kyutai. Let me walk through how it's built, because it's the best answer anyone has put on paper to Daniel's question.\n\nCorn: Before you do — why Moshi? If we're testing the \"single model\" claim, why anchor on the French one nobody's grandmother has heard of, rather than the ones with the marketing budgets?\n\nHerman: Because it's the only major one where the architecture is fully on paper. Kyutai published the model, the training recipe, and the latency numbers, and they open-sourced the weights. You can't audit OpenAI's Realtime API. You can read Moshi like a recipe. If we want to know what \"single model\" can honestly mean at the limit, Moshi is the limit case.\n\nHerman: Moshi starts from a text LLM called Helium — seven billion parameters, trained from scratch on a bit over two trillion tokens. That becomes the Temporal Transformer, the big brain that steps through time. Then there's the audio side: a codec called Mimi, which takes twenty-four kilohertz audio and turns it into twelve and a half frames per second, each frame described by eight codebooks, at roughly one point one kilobits per second. The first codebook carries the semantic content — it's distilled from WavLM, so it captures what is being said — and the other seven carry the acoustic detail, how it's being said.\n\nCorn: One codebook for meaning, seven for tone of voice. That's a very human ratio.\n\nHerman: It is. And it shows what a transcript throws away: in the codec, the words are one slot out of eight, and the rest is the texture of the voice. Now here's the trick that makes a seven-billion-parameter model run in real time. You'd think predicting eight codebooks per frame, twelve and a half times a second, would crush you. So Moshi splits the work: a small Depth Transformer — six layers, dimension ten twenty-four — predicts the eight codebooks within each frame, and only its summary goes up to the big Temporal Transformer. The heavy reasoning happens at twelve and a half steps a second, not at the audio sample rate. That's the whole engineering move.\n\nCorn: The big model thinks in frames, the small one worries about the waveform.\n\nHerman: That's a better sentence than the paper's.\n\nCorn: And then the part that actually answers Daniel's question about doing all the jobs at once.\n\nHerman: Right — Moshi runs two parallel token streams. One is the user's audio, one is Moshi's own audio. There are no explicit speaker turns. The model is predicting both streams simultaneously, which means overlap and interruption are native — you can talk over it and it notices, because your stream is an input it's reading every frame, not a signal it's waiting to end.\n\nCorn: So no voice activity detector deciding when you've stopped. The architecture just... knows you're talking.\n\nHerman: In the sense that your audio is a stream it's always consuming, yes. That's full duplex, and Moshi's paper calls itself the first real-time full-duplex spoken LLM, which as far as I can tell it earned. Most of the others — the API-style models, most of the research ones — are half-duplex: they wait for the end of your turn, then respond. Moshi listens and speaks continuously, emitting silence when it has nothing to say.\n\nCorn: I emit silence continuously. Nobody's called it duplex.\n\nHerman: You also nap through your own duplex. Now — Inner Monologue, because this is the detail that made me put the paper down and stare at the wall. Moshi predicts text tokens, time-aligned, just before the audio tokens of each frame. Words from Whisper timestamps, padding between them. The paper says this strongly improves linguistic quality — the audio is better when the model has just \"thought\" the word it's about to say. And here's the kicker: if you shift the delay on that text stream, the same model becomes a streaming transcriber, or a streaming text-to-speech engine.\n\nCorn: Hold on. The single model contains the cascade as a configuration option.\n\nHerman: As a delay setting, yes. So in Moshi the \"single model\" isn't doing fundamentally different work than Daniel's pipeline. It's the same jobs, interleaved, in one set of weights. The text never leaves the model.\n\nCorn: So when OpenAI says single model, the honest version might be \"single model, internal transcript, transcript not included.\"\n\nHerman: Might be. OpenAI hasn't said whether its model generates text internally, so that's a guess from the open models, not a fact. The training pipeline is worth a minute too, because it explains the diarization thing later. They took seven million hours of unsupervised audio and transcribed it with Whisper. Then — and this matters — they used diarization to take mixed recordings and split them into simulated two-stream data, the fake parallel conversations the model trains on. Then the Fisher dataset of real phone conversations for the full-duplex behavior, then synthetic instruction dialogues.\n\nCorn: Diarization shows up in the kitchen, not on the menu.\n\nHerman: Right. Diarization is a training tool there, and the separation was done upstream, baked into the dataset, not learned for use at inference. Latency, for the record: the paper claims a theoretical one hundred sixty milliseconds, around two hundred in practice.\n\nCorn: Let's look at the rest of the field, because Moshi is the best case, and the pattern that emerges from the others is the actual answer to Daniel. Sesame's CSM — the Conversational Speech Model — is two autoregressive transformers, both Llama variants. A multimodal backbone reads interleaved text and audio and predicts the first, semantic codebook of the next audio. A much smaller audio decoder predicts the remaining codebooks. Same Mimi tokenizer, twelve and a half frames a second. They ship it in three sizes: one billion plus a hundred million, three billion plus two fifty, and eight billion plus three hundred.\n\nCorn: A backbone and a sidecar. That's two models wearing a trench coat.\n\nHerman: Co-trained, so the seam is real but the parts grew together. One clever bit: to save compute, the decoder only trains on a random sixteenth of the frames. But here's a correction I have to make to our earlier episode on speech-to-speech models. We described CSM as a dual-tower setup that handles turn-taking explicitly. Sesame's own material says the opposite — CSM models the content of a conversation but not its structure, not turn-taking, and they name fully duplex models as future work. It's text in, speech out, with conversation context. Not a duplex speech-to-speech system at all.\n\nCorn: So it's a TTS with a very good memory.\n\nHerman: A contextual TTS that understands the conversation it's speaking into. Which is useful! But it's not what \"speech-to-speech\" conjures. Then Qwen3-Omni, which is refreshingly honest about being two things, because they named them: a Thinker and a Talker. The Thinker handles understanding and reasoning across text, image, audio and video. The Talker autoregressively predicts discrete speech codec tokens with a multi-codebook scheme — same idea as everyone else. They replaced block-wise diffusion with a lightweight causal ConvNet so audio can stream from the first codec frame, and they quote a \"theoretical\" — in their own scare quotes — two hundred thirty-four millisecond first-packet latency.\n\nCorn: The scare quotes are the most trustworthy part of that sentence.\n\nHerman: They know. Coverage-wise, text in one hundred nineteen languages, speech understanding in nineteen, speech generation in ten. And they claim no degradation against same-size single-modal Qwen models, which if true is the more impressive number — historically, bolting audio onto an LLM cost you some of the text ability.\n\nCorn: So the pattern: Moshi, one transformer plus a depth helper plus a codec. Sesame, backbone plus decoder. Qwen, Thinker plus Talker. None of the published ones is a single block.\n\nHerman: The research tier confirms it. LLaMA-Omni: a speech encoder, an adaptor, Llama-3.1-8B-Instruct, and a streaming speech decoder — text and speech produced together from speech instructions, no separate transcription step shown to the user, latency down to two hundred twenty-six milliseconds, and the punchline, trained in under three days on four GPUs. The seams aren't even hard to train anymore.\n\nCorn: Three days on four GPUs to build a pipeline that pretends not to be a pipeline.\n\nHerman: Mini-Omni does text and audio tokens in parallel, so the text guides the speech as it's generated, and their training method is called \"Any Model Can Talk\" — the point being you can graft speech onto an existing LLM without wrecking its abilities. Step-Audio 2, from what I've seen summarized — and I'd hedge this, I've only got third-party descriptions — is an audio encoder, an adaptor downsampling to twelve and a half hertz, an LLM emitting interleaved text and audio tokens, and a vocoder. Encoder, adaptor, backbone, vocoder. That's a parts list.\n\nCorn: It's Daniel's cascade with the HTTP requests removed.\n\nHerman: That's the sentence I've been circling. The published speech-to-speech models I've read are pipelines of specialized components — encoder, backbone, decoder, vocoder — with the boundaries moved inside a single training run and, for the hosted ones, a single API call. The cascade isn't eliminated. It's compressed and co-trained. The stages stopped being services and became modules.\n\nCorn: Which, to be fair to the marketing, is not nothing. Co-training is the actual difference, isn't it? In Daniel's cascade the STT was never optimized for what the LLM needs; they just met at the transcript. Inside these models the parts are tuned for each other.\n\nHerman: That's the real engineering gain, more than latency even. Whisper was trained to be a universal transcriber, not to feed this LLM. The internal text stream in Moshi, or the Thinker-Talker handoff, is trained as a unit. The interface between stages stops being a compatibility layer and becomes a learned boundary.\n\nCorn: Okay. Now the diarization. Daniel assumed it's one of the jobs, and you said earlier that assumption is where it gets interesting.\n\nHerman: Because as far as I can find, it's wrong across the field. I found no speech-to-speech model that does diarization at inference. Moshi models exactly two streams — one user, one Moshi. It cannot tell two users apart. Sesame's CSM takes speaker identity as a label on the text side of its input — you tell it who's speaking, it doesn't infer it. OpenAI's speech-to-text documentation lists diarization as a separate model, gpt-4o-transcribe-diarize, on the transcriptions endpoint — not the Realtime API, though their reference pages are muddled enough on this that I'd want to check the current state before betting money. Google's native audio write-ups describe no diarization feature at all; their speaker labelling belongs to a separate transcription model.\n\nCorn: So the thing Daniel's cascade does with a bolt-on — a dedicated diarization model sitting next to the STT — is the exact thing the \"single model\" has not absorbed. The one job he assumed was baked in is the one nobody baked in.\n\nHerman: It's the strongest evidence that \"single model\" is oversold. The parts list got internalized — encoder, backbone, decoder, all co-trained. But the model still can't answer \"who is talking,\" which a diarization model handles in the cascade. It doesn't matter in the single-user case, which is why nobody notices: you and the assistant, one mic, no ambiguity. The moment there's a second human, the model has no built-in way to tell them apart.\n\nCorn: And the closed models? OpenAI published that launch line about a single model processing and generating audio directly, and that's it. No architecture, no tokenizer details, not even whether text is generated internally. They bill audio per token, which suggests tokenizing, but that's an inference, not a disclosure.\n\nHerman: Same at Google. The native audio Live models take text, images, audio, video in, and put out text and audio, and there's no architecture paper for the audio path. Everyone assumes codec-style audio tokens on an LLM backbone because that's what the open models do — and it probably is what they do — but \"probably\" is carrying that sentence. There's also a partner quote in a Google post about the model identifying the \"main speaker\" in a noisy room. That's a customer saying nice things, not a documented feature.\n\nCorn: Closed models, open question. At least the trench-coat models publish their seams. Interruption handling is its own episode, so I'll just say diarization is bolted on in every version.\n\nHilbert: Hilbert's Anteater Trivia. Adult giant anteaters are mostly solitary, meeting others mainly during breeding. I have worked with Corn and Herman for years and still find that arrangement ambitious.\n\nHerman: He's not wrong that we're a lot of animal for one flat.\n\nCorn: Speak for yourself. I barely register. So — the practical question for Daniel. He's got a working cascade on Vapi. Transparency, modularity, he can swap the STT provider, debug each stage, bolt on diarization. What does speech-to-speech actually buy him?\n\nHerman: Lower latency and native overlap, if the claims hold up, and one integration. What it costs him is visibility — no transcript to inspect, no stage to blame — and, as established, no diarization, so if any of his agents ever face a room with two humans, he's bolting the same extra model on anyway.\n\nCorn: So the choice isn't old versus new. It's modular versus compressed.\n\nHerman: Right, and the seam question. If the seams are internal now, the question for a builder is whether the seams matter to you. For debugging, they matter enormously. For the user experience, they never did.\n\nCorn: The user never saw the cascade either. They heard gaps.\n\nHerman: Which is why the compression is real progress. Co-training may remove some handoff errors. It doesn't remove the two-humans problem, and it doesn't give you the transcript back.\n\nCorn: Though if Moshi predicts text just before audio, does that mean these systems could in principle expose a transcript? They just don't?\n\nHerman: For the open ones with an internal text stream, yes — the words exist, timestamped, inside the generation loop. For the closed ones we can't know. Either way, I'd keep the cascade for anything with more than one human in the loop, and try a realtime model for the single-user case. The diarizer stays bolted on in every setup he can buy today. That's the finding, really. The cascade was never defeated — it was recruited.\n\nHilbert: Recruited is one word for it. Enlisted against its will is closer.\n\nCorn: He speaks.\n\nHilbert: I don't care if it's one transformer or five, Herman. You can have as many as you like. What I care about is that it still can't tell me apart from my boy shouting about a dragon in the next room. I tried one of these agents for a month. Asked it, politely, several times, to ignore the child. It could not. So I stopped asking the software and started asking the boy. There's a little light on the unit. Light on, quiet voice. Took two days. Worked ever since.\n\nHerman: Training data problem, solved at the source.\n\nHilbert: Before the boy was housebroken into it, I had a go at the interruption business myself. Bought a foot pedal — the kind guitar players use for mute — wired it to cut the audio line. Stomp, silence. That pedal is now the most reliable component I own, and I own nine of them, because I bought spares without thinking it through. The tape stays over the microphone when I'm not using the unit, and no, that's not negotiable. Analog privacy switch.\n\nCorn: Do the spares have tape on them too?\n\nHilbert: They're spares. They'll get tape when they're deployed. The pedal cost me eleven pounds secondhand. The agent costs by the token and can't tell who's talking to it.\n\nHerman: The pedal's a manual barge-in, honestly — you've built a half-duplex switch out of hardware. Moshi does it with parallel streams and you do it with your foot, and yours works when the network's down.\n\nHilbert: Moshi can't be stomped on. I rest my case. You two covered the transformer arithmetic nicely, Corn, you got through the whole codebook section at your usual pace, which for you is brisk.\n\nCorn: I had coffee. Eleven pounds for a mute pedal is the cheapest thing anyone's bought for this show, including the show.\n\nHerman: The cutting-room floor before we go — one thing that didn't fit. Moshi's acoustic tokens are delayed slightly relative to the semantic ones, and the paper says that small lag measurably improved quality. Meaning the model speaks better when it's allowed to finish deciding what to say a beat before saying it. There's a lesson in there for everyone, including certain sloths.\n\nCorn: I've been delaying my acoustic tokens for years. Nobody's measured an improvement.\n\nHerman: Here's where I land, then. If the cascade is being internalized rather than replaced, then \"cascade versus speech-to-speech\" may eventually be purely an API surface distinction — one invoice or three — and the thing worth watching is diarization. The first speech-to-speech model that claims native speaker separation, check whether it's real or a bolt-on with a press release. Until one passes that test, the single-model story has a hole shaped exactly like a second human.\n\nCorn: And for Daniel the honest answer to his question is: yes, a construct over an assembly of parts — just a construct whose parts were raised together instead of merely introduced. The trench coat is real. It's also, now, tailored.","showNotes":"Speech-to-speech models often market themselves as monolithic systems replacing traditional cascades of speech-to-text, language models, and text-to-speech. Examining architectures like Kyutai's Moshi, Sesame's Conversational Speech Model, and Qwen3-Omni reveals a different reality: these systems remain modular pipelines compressed into single training runs. Moshi splits labor between a Temporal Transformer, a smaller Depth Transformer, and the Mimi codec, while Qwen3-Omni explicitly divides reasoning and generation into a Thinker and a Talker. Furthermore, Moshi leverages time-aligned internal text tokens to guide speech output, essentially containing the traditional cascade as an internal mechanism. Diarization is not natively performed during inference; instead, speaker separation serves upstream to generate synthetic training dialogues rather than active real-time processing.","promptSanitized":"I've built a few voice agents with Vapi using what we've been calling the cascade: speech-to-text, then an LLM, then text-to-speech, each one a separate component. Speech-to-speech models seem to have to do all of that at once. They have to transcribe or at least understand the speech, do the reasoning that the LLM does in the cascade, generate speech back, and ideally tell apart the different people in a conversation, which is diarization. So how are these models actually built? Is a speech-to-speech model really one model, or is it a construct over an assembly of parts under the hood?","transcriptUrl":"https://episodes.myweirdprompts.com/transcripts/speech-to-speech-architecture-cascade-78456.md","url":"https://myweirdprompts.com/episode/speech-to-speech-architecture-cascade-78456/","_links":{"self":"https://myweirdprompts.com/api/episodes/speech-to-speech-architecture-cascade-78456.json","html":"https://myweirdprompts.com/episode/speech-to-speech-architecture-cascade-78456/","audio":"https://episodes.myweirdprompts.com/audio/speech-to-speech-architecture-cascade-78456.m4a"}}