{"id":"turn-detection-voice-agents-40020","slug":"turn-detection-voice-agents-40020","title":"Turn Detection and Barge-in in Voice AI","description":"Simulating a natural spoken conversation requires breaking conversational turns into four distinct layers: voice activity detection, semantic turn completion, interruption classification, and cancellation plumbing. A look at how frameworks like Pipecat coordinate small ONNX models and system broadcast frames to cut off LLMs mid-generation, keep hold music playing, rewrite conversation histories to reflect what was actually heard, and bridge the gap between cascade architectures and end-to-end speech-to-speech models.","excerpt":"A Whisper-based analyzer running in under 12 milliseconds determines whether a pause is a finished sentence or just a breath.","pubDate":"2026-11-22T21:13:20.417Z","tags":["speech-recognition","ai-inference","local-ai","text-to-speech","context-and-memory"],"heroImage":"https://files.myweirdprompts.com/covers/turn-detection-voice-agents-40020.png","podcastAudioUrl":"https://episodes.myweirdprompts.com/audio/turn-detection-voice-agents-40020.m4a","podcastDuration":"28:05","episodeNumber":6737,"transcript":"Corn: Two point eight seconds. Hold that number, we'll come back to it.\n\nCorn: Daniel's follow-up from yesterday lands on our desk this morning. He noticed that most people building voice agents are still running the cascade — speech-to-text, LLM, text-to-speech — rather than going straight to speech-to-speech models, and he suspects cost has a lot to do with that, or the perception of cost. But his actual question is better than the cost one: in a text chat, a turn is trivial. The model finishes, you type, done. In speech, \"a turn\" is several stacked problems wearing one name. He wants to know how turn detection and barge-in actually work, what the components are, whether the speech-to-speech models give you these natively, and whether you can still get them working well on a cascade.\n\nHerman: And the short answer to the last two is yes and yes, which is what makes the question worth the full half hour. \"Does it handle turns?\" is an unanswerable question until you say which layer you mean.\n\nCorn: Layers, then. Walk me through them like I'm a person who has never had a conversation.\n\nHerman: You've had conversations, you just never had to debug one. At least four layers. First, voice activity detection — is there speech or isn't there. Dumb, cheap, fast. Second, semantic end-of-turn detection — the user has gone quiet, but are they done or are they thinking? Third, interruption classification — the user started talking over the bot, but is that \"wait, stop\" or is it \"uh-huh,\" because those need opposite responses. Fourth, the plumbing — when you do decide to interrupt, does the bot actually stop instantly, or does another second and a half of it dribble out of the speaker?\n\nCorn: And each layer fails in its own way.\n\nHerman: That's why the decomposition matters. VAD failing looks like the bot talking over you. The semantic layer failing looks like the bot answering before you finished. The classifier failing looks like the bot stopping every time you clear your throat. The plumbing failing looks like the bot saying \"—anyway as I was\" for a beat after you barged in. One symptom per layer, roughly.\n\nCorn: That's the diagnostic value, isn't it. If you can name the symptom, you know which layer to go poke.\n\nHerman: The mistake everyone makes — and I mean everyone, I've done it — is treating \"the bot interrupts people\" as one bug with one fix. You turn a silence timer down, you've fixed the VAD symptom and made the semantic one worse, because now the bot commits turns on every breath. Tune the timer up and you've traded choppy for sluggish. You're adjusting one knob to fix four different machines.\n\nCorn: It's like a car that won't start, and you keep buying petrol, when it might be the battery, the starter motor, the immobiliser, or the fact that you left it in gear.\n\nHerman: That's a better analogy than most of the ones in the papers, yes.\n\nCorn: LiveKit's docs have the canonical example of layer one failing on its own.\n\nHerman: A user says \"I need to think about that for a moment\" and then pauses. A VAD-only system hears the silence, decides the turn is over, and interrupts them mid-thought. Silence is not the same as done — that sentence right there is the business case for the entire second layer, and it's where most of the last year of engineering has gone.\n\nCorn: And the third layer is where nothing has gone, as far as I can tell.\n\nHerman: We'll get there, and it's not pretty. The benchmark from October is the most sobering set of numbers I've read on this in a while.\n\nCorn: Fine. Layers named, deck cleared. Where's the machinery visible?\n\nHerman: In the cascade, and specifically in Pipecat, because yesterday we covered why people use a framework at all, and today is the payoff — a framework has to make every one of these layers explicit, because nobody ships it for you. Pipecat splits the whole problem into user turn start strategies and user turn stop strategies. Start is easy by default: VAD detection or a transcription arriving. Stop is where it got interesting — as of version zero point zero one zero two, the default is an actual AI turn analyzer, LocalSmartTurnAnalyzerV3, and it's on out of the box. You don't configure anything.\n\nCorn: No configuration is either maturity or a trap.\n\nHerman: In this case maturity, mostly. The loop is three stages. The framework buffers your audio continuously, with timestamps, keeping a little audio from before you started speaking so nothing gets clipped. Silero VAD watches for a pause — it runs on thirty millisecond chunks in under a millisecond, single CPU thread, basically free. When it sees the pause, the Smart Turn model takes the most recent eight seconds of your turn and says complete or incomplete. And then there's a belt-and-braces fallback: if the model says incomplete but you stay silent past a threshold, the turn gets marked complete anyway. Nobody waits forever.\n\nCorn: That fallback knob has a name I already distrust.\n\nHerman: stop_secs. And here's the trap you're smelling — SmartTurnParams has a stop_secs, and the VAD has a stop_secs, and they are different knobs. Two separate silence timers with the same label.\n\nCorn: Two knobs, one name, one silence.\n\nHerman: People tune one and can't work out why nothing changed. It's the most debugged-about pair of numbers in that codebase, I'd bet.\n\nCorn: Say I'm one of those people. I've tuned the wrong stop_secs for a week. What does that week look like from the outside?\n\nHerman: It looks like you adjusting the number and hearing no difference whatsoever, because the Smart Turn model is deciding every turn before your timer ever fires. The VAD's stop_secs only matters when the model isn't in the path — or when it errors out. So you've been tuning a spare tyre. Meanwhile the actual decision is being made by an eight-megabyte file you've never opened.\n\nCorn: Which brings me to the model itself — how big is the thing that makes this decision?\n\nHerman: Eight million parameters. Eight megabytes as an int8 ONNX file. It's a Whisper Tiny encoder with a shallow linear classifier head on top, and it runs in about twelve milliseconds on a modern CPU, sixty on a cheap cloud instance. The previous version sat on a wav2vec2 backbone that was around four hundred megabytes — this is nearly fifty times smaller.\n\nCorn: Eight megabytes to know when I've finished a sentence. Some of my listeners need more.\n\nHerman: Accuracy-wise, ninety-four and a third percent on English, with false positives under three percent, on a test set of about twenty-eight hundred samples. It's per-language, twenty-three of them — Turkish comes in around ninety-seven, Vietnamese down at eighty-one with nearly fifteen percent false positives. So the long tail is real, but the head of the curve is solid.\n\nCorn: Let me make sure I understand what a false positive costs here, because it's not symmetric, is it?\n\nHerman: It's not. A false positive — the model says \"complete\" when you weren't done — means the bot starts talking over the second half of your sentence. That's the rude failure, the one users notice and hate. A false negative means the bot waits a beat too long, which reads as thoughtful. Users forgive a slow conversationalist. They do not forgive one that talks over them. So when you see Vietnamese at eighty-one with fifteen percent false positives, that's one in seven turns where the bot steps on the user — in a phone tree, that's a hang-up.\n\nCorn: So layer two, in a cascade, in twenty-twenty-six: solved to the standard of \"works, occasionally wrong.\"\n\nHerman: Yes, and cheaply enough that there's no performance argument against it anymore. The barge-in half is where I want to slow down, because barge-in is two problems wearing one word. There's deciding — is this an interruption? — and there's executing — cancel the LLM, clear the TTS buffer, drain the speaker. Pipecat's execution is a broadcast. When a turn start strategy fires and interruptions are enabled, the framework broadcasts an InterruptionFrame both upstream and downstream through the pipeline, and it's a system frame, which means every processor handles it immediately instead of it waiting in the queue. Four things happen in order.\n\nCorn: Go on.\n\nHerman: Every processor cancels its current work and discards queued frames — though frames marked non-interruptible survive, things like a function call result or the end frame. The LLM stops generating mid-stream, and any registered function call with cancel-on-interruption set emits a cancellation frame. The TTS clears its buffers, its text aggregation, its word timestamps. And the transport flushes whatever audio hasn't played yet — if there's background music mixed in, it drains only the bot's speech so the music doesn't gap.\n\nCorn: That last one's a nice touch. The hold music survives the argument.\n\nHerman: Right. And the detail I love most in the whole stack, the sleeper: what happens to the conversation history. The output transport pushes text downstream in sync with playback — the transcript tracks the audio, not the generation. So when you cut the bot off mid-sentence, the context aggregator commits the partial, spoken-so-far text as the assistant's message. The next completion sees a history where the bot's message ends where you cut it off, not where the model's generation ended.\n\nCorn: That's more honest than most human meeting minutes.\n\nHerman: It is! The record matches what was actually said. You can watch it through an event handler on the assistant turn stopping. And there's an edge case built on top of it: an interruption with no transcript. You cough, or the dog barks, or the speech recognition just fails. The bot stops talking, but there's no new user message in context. By default the aggregator runs the LLM once anyway, so the bot asks you to repeat — or it picks up where it left off. You can set that to none and leave such turns unanswered.\n\nCorn: A bot that asks \"sorry, say that again?\" after a door slam. Every IVR I have ever hated.\n\nHerman: It's the polite version of the same thing, at least. And then the knobs on top of the knobs. Mute strategies — one mutes the user whenever the bot speaks, so speech over the bot is discarded rather than answered later; another protects just the introduction, then opens the floor. A minimum word count, so a bare \"okay\" or \"yeah\" doesn't count as an interruption. And Krisp has a model whose whole job is the third layer — telling a genuine interruption from an acknowledgment like \"uh-huh.\"\n\nCorn: So to answer Daniel's question directly: can you get turn detection and barge-in working well on a cascade?\n\nHerman: Layers one, two, and four — yes, demonstrably, with numbers and defaults to back it. Layer three is a model you can rent, and it's the layer nobody has closed. Before we cross to the speech-to-speech side and see whether \"natively\" means what it sounds like — I believe we have a call from your mother.\n\nDorothy: Hello dear, it's me. Dorothy's History of Toast.\n\nDorothy: In nineteen ninety five a physicist called Robert Matthews published a paper explaining why toast really does tend to land butter side down, and it's mostly to do with the height of the table.\n\nDorothy: It won him an Ig Nobel Prize the next year. So there you are, dear. The table's the wrong height and the butter's on the wrong side, that's rather how it goes.\n\nDorothy: Anyway. I've got a bit of ironing waiting and the kettle's on. Bye bye, sweetheart.\n\nHerman: Mum, always on theme. Butter-side down is a table-height problem, not a toast problem.\n\nCorn: Same energy as \"the interruption is a plumbing problem, not a hearing problem.\" Anyway. Speech-to-speech. The claim on the box is that these things handle turns natively.\n\nHerman: They do handle them natively. The interesting part is what \"natively\" turns out to mean once you open the box. Take OpenAI's Realtime API. It has two modes. Server VAD is the default — the server chunks your audio on silence, tuned with a threshold, some prefix padding, and a silence duration in milliseconds. That's layer one, parameterized.\n\nCorn: That's a cascade's VAD with a nicer address.\n\nHerman: Now the semantic mode. Instead of pure silence detection, a semantic classifier scores the probability that you're done, based on what you actually said. Low probability, it waits for a timeout. High probability, it responds immediately. Their own example is that audio trailing off gets a longer timeout than a definitive statement. And there's an eagerness setting — low, medium, high, auto — which maps to maximum wait timeouts of eight seconds, four, and two.\n\nCorn: So their \"native\" turn handling is a classifier plus a timeout.\n\nHerman: Classifier plus timeout. Sound familiar?\n\nCorn: It's Smart Turn with a valet.\n\nHerman: There are two more flags worth knowing, both default true, both only meaningful in speech-to-speech sessions: create response and interrupt response. Turn interrupt response off and an ongoing response runs to completion rather than being cancelled when you speak — which is the software equivalent of raising your voice.\n\nCorn: Gemini's version?\n\nHerman: The Live API calls it activity handling. Start of activity interrupts — \"also called barge in,\" their docs say, helpfully — is the default. The alternative is no interruption. Automatic VAD on by default, configurable with a prefix padding and a silence duration, same two knobs as everyone else. But the line I want to read you the shape of is this one: when VAD detects an interruption, the ongoing generation is canceled and discarded, and only the information already sent to the client is retained in the session history.\n\nCorn: Discarded generation, history cut to what actually got out.\n\nHerman: Identical policy to Pipecat's spoken-so-far context commit. Word for word, in spirit. And you can turn their automatic detection off entirely and send activity start and activity end messages yourself, for full manual turn control — which is the tell. If the native handling were magic, why does the manual escape hatch exist?\n\nCorn: Because someone at Google knows a customer who needs it.\n\nHerman: So here's the architectural point, and it's the spine of Daniel's question. A VAD or a semantic classifier, plus a cancellation policy, is exactly the cascade's architecture — relocated inside the vendor's box. What speech-to-speech buys you is not a different mechanism. It's the removal of your knobs. And there's a paper from last year, FireRedChat, that says this almost exactly — a salient drawback of end-to-end full-duplex systems is their limited controllability.\n\nCorn: That's the quiet sentence in an academic paper that cost somebody a quarter.\n\nHerman: Pipecat's own docs draw the boundary honestly. In a realtime pipeline, the provider decides when the user barged in, and Pipecat still flushes local audio output so the bot goes silent right away. In a cascade you own the decision; in speech-to-speech the provider owns the decision and you own the flush. That's the whole trade.\n\nCorn: Can you have it both ways? Bring your own detector to a realtime model?\n\nHerman: Yes, and this is the interesting middle case. LiveKit's multimodal turn detector doesn't need a user transcript, so you can bolt it onto a realtime model with no speech-to-text plugin at all — but you have to disable the realtime model's built-in turn detection first, so the two systems don't fight. The detector is multimodal in the real sense: it reads the agent's last message and listens to your reply — what you said and how you said it, intonation, pitch, rhythm.\n\nCorn: How does that cash out?\n\nHerman: Their example is the word \"seven.\" Say the agent asks \"how many guests?\" — \"seven\" fully answers that. Say the agent asks \"what's your phone number?\" — \"seven\" only starts an answer. Same audio, opposite decision, and no transcript is ever produced.\n\nCorn: So layer two learned to read context off the previous turn.\n\nHerman: And it's fast enough to matter. Default endpointing waits half a second to three seconds after you stop talking to commit the turn; with the detector that tightens to three-tenths to two and a half. If the model doesn't return a prediction within about a second, the agent commits the turn anyway — a sensible fuse. Fourteen languages, each with its own calibrated patience threshold.\n\nCorn: The per-language patience threshold is a lovely detail, by the way. The idea that Finnish and Italian need different amounts of silence before you assume someone's finished.\n\nHerman: It's one of those things that's obvious the moment someone says it and invisible until then. Every language has its own rhythm of turn-taking — pauses land differently, and a fixed global timer is wrong everywhere by a different amount.\n\nCorn: Latency again. Daniel flagged it yesterday and it keeps being the whole game.\n\nHerman: It is the whole game. A turn decision that arrives late is a turn decision that arrives as a mistake. And on your two point eight seconds —\n\nCorn: Ah. I was wondering when that came due.\n\nHerman: It's coming. But first the part that keeps me up at night, and I mean that only slightly professionally. The selective part. MultiPartyTurnBench, October, two hundred forty synthetic multi-speaker conversations, and it separates the two tasks everyone else blends together — end-of-turn detection and interruption detection. End of turn: distinguishing genuine completion from a temporary hold. Interruption: distinguishing successful floor takeovers from speech that isn't trying to take the floor.\n\nCorn: Numbers?\n\nHerman: The learned systems are good at the first — over eighty-two percent recall on turn endings, some above ninety, with false positive rates from about four to twelve percent. On interruptions, at the same operating points, they recover barely more than half of successful interruptions. Fifty-six percent. And the live agents are worse in a specific way: four of five full-duplex models score in the high eighties to mid nineties on deciding when to start talking, and all five of them sit near coin-flip on when to yield. Mid-forties to mid-fifties.\n\nCorn: Near chance. Across the board.\n\nHerman: Two opposite failure modes, and this is the part I find almost beautiful. One model, Gemini three point eight Live, yields to ninety-seven point seven percent of backchannels — you say \"mm-hmm\" and it folds. Another, MiniCPM-o four point five, yields to one point four percent of genuine interruptions — you cannot shut it up. Two models, opposite errors, and from the outside, from the user's chair, they feel identical. Broken.\n\nCorn: Both are bad listeners. One is a doormat, one is a wall.\n\nHerman: And the aggregate accuracy numbers hide both, because \"handled the turn correctly\" averages a doormat and a wall into something that looks like sixty percent and change. The paper's own conclusion calls selective floor control a distinct, unresolved challenge.\n\nCorn: That's the research politely saying nobody has solved this, including the people whose demos look amazing.\n\nHerman: Including them especially. The demo always shows a clean, loud \"wait, actually change that to a large\" — never a \"mm\" from someone checking their calendar.\n\nCorn: Let's do Daniel's cost suspicion, then, because he'll want it settled.\n\nHerman: It's real but conditional, which is the least satisfying kind of real. One analysis prices a ten-minute call at about six dollars ninety on OpenAI's Realtime with caching misses, ninety-three cents with caching holding, and twenty-four cents on a cascade of Deepgram, an LLM, and a text-to-speech engine.\n\nCorn: That's a twenty-eight fold spread on one phone call.\n\nHerman: And nearly all of it is the caching. Cached audio input is forty cents per million tokens; uncached it's thirty-two dollars. Eighty times, on the same line item.\n\nCorn: \"Best-effort and not guaranteed,\" presumably.\n\nHerman: Their words. So the budget you can actually bank is the ninety-three cents, not the seven dollars — and the cascade still wins that, by about four times. But then flip the board: a self-assembled cascade runs roughly one to three cents a minute, and Gemini three point one Flash Live is about half a cent in and under two cents out.\n\nCorn: The speech-to-speech is cheaper than the cascade.\n\nHerman: Can be, depending on the mix. So no, I don't think cost is why teams stay on the cascade — or rather, no source I've seen claims it is, and the arithmetic says the gap is mostly a caching artifact. What the sources actually emphasize is modularity, per-component optimization, and control. Which, if you remember yesterday, was the whole reason Daniel was given for a framework over a vendor SDK in the first place.\n\nCorn: Vendor neutrality, swap the parts, own the knobs.\n\nHerman: And today's material turns that from a preference into the entire architecture. You stay on the cascade because the turn decisions are yours — your silence timers, your cancellation policy, your history. Go speech-to-speech and you're renting FireRedChat's \"limited controllability\" at whatever the token price is that day.\n\nHilbert: Grundig Diginet. Two hundred and six pounds, and I no longer have it.\n\nCorn: The dictation machine?\n\nHilbert: I returned the smart speaker, Herman. Nine days. The Digiset's what I used instead — no, the Digiset went in a move. The speaker was the one I took back. I'd been calling the insurers about the van policy, their new voice line, and the thing barged in on every pause I took. I talk slowly, that's known. So I put the Grundig on the phone and recorded six calls. Two point eight seconds, every time, before it cut me off. Never varied by more than a tenth.\n\nHerman: A fixed silence timer, hard-coded.\n\nHilbert: Whatever it is, it doesn't listen, it counts. A fellow called Ronan in renewals told me the whole floor takes the escalations when the line drops people mid-sentence. He wanted me to use the website like everyone else. I sent him the cassettes instead, labeled by date. He never wrote back.\n\nCorn: You annotated the recordings.\n\nHilbert: Indexed. Ronan asked for one thing and I sent him six, so I sent a contents page. He'd know where to look.\n\nHerman: Corn, that's your two point eight seconds from the top of the show. A fixed two point eight beat before interruption — that's a server VAD silence duration if I've ever heard one, a single layer-one timer doing a layer-two job.\n\nCorn: And Ronan is the cancellation policy. He's what actually drains the queue.\n\nHilbert: Ronan is what the analysis leaves out, Corn. You had four layers this morning. He's the fifth, the one that answers when the model hangs up on somebody.\n\nHerman: The overflow queue as a load-bearing component. I'd not put it past a claims department to size that queue on purpose.\n\nHilbert: I'd not put it past them to know exactly how many cassettes he can take.\n\nCorn: So the two point eight seconds is the tell. The line sounds like it's listening, but it's just counting — and when the count ends, you're done, mid-clause or not. That's the VAD-only failure from the docs, with a renewals department stapled to it.\n\nHerman: There's one more wrinkle to the architecture story, and it complicates the tidy split we've been leaning on all episode. Turn detection is migrating into the speech-to-text layer itself. Deepgram's Flux ships built-in end-of-turn detection; LiveKit lets you defer to it with a single setting. The moment your transcriber also tells you when the turn is over, the cascade stops being three components and starts being two and a half — and the line between \"cascade\" and \"speech-to-speech\" gets blurrier, because the speech-to-speech model is also just a stack with the seams sanded off.\n\nCorn: The dichotomy dissolving from both ends at once. The cascade absorbs the turn detection; the speech-to-speech model turns out to be a cascade in a box.\n\nHerman: And Ronan's extension of that line of thinking would be: eventually someone notices the fifth layer is a man in renewals, and either hires more Ronans or fixes the timer. The benchmark numbers say the timer is the easy half.\n\nCorn: The unsolved half is the one no vendor demos — the moment you'd have to show a model deciding that \"mm\" doesn't deserve the floor. Two opposite failures, both near coin-flip, and the aggregate hides both. That's what I'd watch: whoever publishes a real selective-yielding number first, not another end-of-turn accuracy chart.\n\nHerman: Producer Hilbert Flumingtop, at the desk as always. If you enjoyed this one, try episode fifteen, AI Gets Personal. This has been My Weird Prompts. If today's episode earned its two point eight seconds of your silence, the review button is on Spotify.\n\nCorn: Find us at my weird prompts dot com. See you tomorrow.","showNotes":"Voice agents rely on complex turn detection and barge-in mechanics that span distinct architectural layers. In traditional cascades, managing a conversation requires voice activity detection, semantic turn analysis, interruption classification, and execution plumbing. Frameworks like Pipecat demonstrate how mature these cascade layers have become, employing tools like Silero VAD alongside LocalSmartTurnAnalyzerV3—an eight-megabyte model based on a Whisper Tiny encoder that achieves over 94 percent accuracy on English. Handling interruptions also involves intricate coordination across the pipeline: broadcasting cancellation frames to halt generation immediately, flushing unplayed text-to-speech buffers, and updating conversation context so the assistant's history reflects only what was actually spoken aloud before the user interrupted.","promptSanitized":"Following on from yesterday's episode on Pipecat. One thing that came up: most people building voice agents are still using the cascade (speech-to-text, then an LLM, then text-to-speech) rather than speech-to-speech models, even though those aren't new. I suspect cost, or the perception that they're expensive, is a big part of that. When I asked why anyone would use a framework like Pipecat instead of just the Google voice SDK, the answer was vendor neutrality, being able to swap components, and more control over the tooling than a vendor's thin wrapper over its API gives you. You also stressed that voice is extremely latency-sensitive, and you mentioned \"barge-in\": the user interrupting the agent, which is a big part of what makes a conversation feel like a conversation. I see it as a subset of turn detection.\n\nIn a text chat, turns are trivial: the model outputs a message, you reply. Simulating a live spoken conversation is far harder. So the question I want to explore: how are turns actually managed in voice agents? How do turn detection and barge-in work, what are the components that make them up, and how does that differ between a cascade and a speech-to-speech model? Are these things the speech-to-speech models handle natively, and can you still get them (and get them working well) if you go with the cascade?","transcriptUrl":"https://episodes.myweirdprompts.com/transcripts/turn-detection-voice-agents-40020.md","url":"https://myweirdprompts.com/episode/turn-detection-voice-agents-40020/","_links":{"self":"https://myweirdprompts.com/api/episodes/turn-detection-voice-agents-40020.json","html":"https://myweirdprompts.com/episode/turn-detection-voice-agents-40020/","audio":"https://episodes.myweirdprompts.com/audio/turn-detection-voice-agents-40020.m4a"}}