Here's the question that's been rattling around my head since Daniel sent this in. If you've got four small models running in a chain on a mid-range Android phone, and everything feels smooth, how do you actually know it's working? Or is the phone just unusually idle?
That's the whole episode right there.
It is. Daniel's been using voice-to-text almost exclusively for about two years now, with the goal of fully replacing typing. And he's excited, excited, because he moved from Whisper to Parakeet V2 for on-device speech to text on Android and says it's a massive step up. He's using the 0.6 billion parameter model, five-point quantization, downloaded from Hugging Face. And he found V2 beat V3, which is its own little mystery we'll get into.
The English-only thing.
Right. But here's the part he actually wants to talk about. He says ASR is probably the most active space in voice technology right now, and the models are great, but they're often a partial product. They give you a raw transcript and stop. So we're finally seeing a mature set of small models specifically for post-transcription cleanup, which is really a bundle of different tasks. Punctuation restoration. Ignoring inferred comments, which means if you're dictating and then you say something to somebody else in the room, the model has to infer that wasn't meant for the transcript. Diarization. And paragraphing, which Daniel calls the great unloved part, the thing that evades the attention of even punctuation models, and which he thinks non-semantic heuristics can't solve.
He's right about that.
He wants to benchmark a chained pipeline. VAD, ASR, one or more cleanup models, ideally on his OnePlus Nord 3, which he's careful to note is not a flagship, and also on his usual machine and his laptop. Measuring RAM at every stage of a single dictation. Because feeling that inference is smooth is not the same as knowing it. The phone might just be idle. And he wants to know whether you can even run VAD and an ASR model concurrently before passing to a cleanup model, and then another one, and whether anyone has traced the resource usage of each constituent element.
So let's start with what's actually in the chain, because it's more than one model.
Go.
The way to think about it is at least three stages. Stage one is voice activity detection. It's the bouncer. It decides what's speech and what's silence or noise, so the ASR model isn't burning compute on the sound of your refrigerator. Stage two is the ASR model itself, which produces the raw transcript. Stage three is the cleanup, which is one or more models turning that raw transcript into something a human would actually want to read. And the crucial thing, the thing Daniel's pointing at, is that the ASR model is a partial product by design, not by failure.
It's a concession from the people building it. Microsoft's CoreAI group put out an on-device streaming ASR report in April, and they say it outright. Their evaluation doesn't cover inverse text normalization, which is numbers, dates, currencies. It doesn't cover speaker diarization. It doesn't cover code-switching or custom vocabulary adaptation. That's a frontier lab saying, here's our model, and here's the list of things it doesn't do.
So the model that everybody's benchmarking is one component of a larger system that nobody's benchmarking.
And the cleanup taxonomy is worth naming concretely, because Daniel lists them and they're different problems wearing similar clothes. Punctuation restoration. Disfluency removal, so the ums and the false starts. Inverse text normalization, which is turning "twenty three dollars and fifty cents" into the numeral form. Diarization, who said what. Turn detection, which is where the ignoring-inferred-comments thing lives. And paragraphing.
And the asymmetry that organizes this whole episode is that punctuation restoration is well-studied, and paragraphing is basically absent from the literature. Daniel says it's unloved. I went looking for a dedicated paper on paragraph segmentation for dictation and found nothing. Searches came back empty. That's not a gap in my search technique, that's a gap in the field.
The silence is the evidence.
The map is clear enough. The question is why the cleanup stage is so much harder than it looks, and that starts with VAD.
VAD looks like a pre-filter. It looks like the trivial part. And it isn't, because the choice of VAD changes what the ASR model even sees. WhisperPipe, which is a streaming architecture paper from April, combines Silero VAD with energy-based filtering, and that combination cut false activations by thirty-four percent. That's not a latency improvement, that's a quality improvement. Fewer false activations means fewer phantom segments for the ASR model to hallucinate words into.
So the bouncer isn't just deciding who gets in. The bouncer is deciding what the band plays.
That's the shape of it, yes. And it gets worse, because the real problem is turn detection. This is the ignoring-inferred-comments task Daniel described. If you're dictating and you turn to your kid and say "put your shoes on," the pipeline needs to know that wasn't part of the document. And the obvious heuristic is silence. A long pause means the turn ended, right?
It does not mean that.
It does not. There's a paper from July, "The Trade-off Was in the Labels," that shows within-turn pauses routinely exceed between-turn gaps. So if you're a VAD with a silence timeout, you cannot separate them. The pause inside a thought is often longer than the pause between two thoughts. What distinguishes them is whether the words so far form a complete thought. That's it. That's the only signal that works.
So the acoustic information is actively misleading.
Actively. And their fix is elegant. They train a small LoRA adapter on Qwen3-ASR-0.6B, hours on one GPU, and it transcribes and detects end-of-turn from meaning. Ninety-seven percent boundary recall at a median of zero point three nine seconds latency, with zero point three false fires per speech-minute.
Point three nine seconds. That's the number that matters, because that's the delay between you finishing a thought and the system knowing you finished it.
And paragraphing is the same problem in a different costume. Daniel's right that non-semantic heuristics don't cut it. Pause length, token count, time gaps. They all fail for exactly the reason the silence timeout fails at turn detection. They can't see semantic completeness. A paragraph bounds a thought. If your heuristic can't see thoughts, it can't see paragraphs.
And I'll say the thing Daniel said, because he's right and it deserves saying plainly. Paragraphing is a stylistic feature. How you break your paragraphs is part of how you write. It's not decoration on top of the content, it's part of the content. He agreed with the linguist we had on who argued exactly that, that paragraphing bounds your thoughts. And a model that's just counting tokens is not doing style, it's doing layout.
There's a principle in the turn-detection paper that I think is the technical version of what you just said. Every streaming-decision label has to be computable from the input up to the decision point. That's the causal supervision principle. If your label depends on information you don't have yet at the moment you have to make the decision, you're going to oscillate. You'll put a paragraph break here, then the next word arrives and now the break was wrong.
So the naive heuristic doesn't just fail, it fails noisily. It flip-flops.
Right. And FastTurn, another paper from this year, is the honest framing of the whole design space. It unifies acoustic and streaming semantic cues for turn detection, and it says the quiet part out loud. Pure VAD approaches lack semantic understanding. Pure ASR approaches introduce latency. You're not looking for the right answer, you're picking a point on a tradeoff curve.
Which brings us back to Daniel's actual empirical finding, which I want to sit with, because it's not a fluke. He found Parakeet V2 beat V3. Same family, newer version, worse result for him.
And the quantization data explains why. Quantization tolerance is model-specific. Nemotron's int4 variant barely moves, eight point zero three percent word error rate down to eight point two zero. Parakeet's int4 goes to eleven point five zero percent. Same quantization scheme, wildly different damage.
So when Daniel says he found the five-point quantization that doesn't challenge his hardware, he's describing a property of Parakeet specifically. He got lucky in the sense that his model tolerates what he did to it. A different model at the same compression would have fallen apart.
And V2 versus V3 fits the same story. V3 is multilingual, spreads its capacity across languages, and for a single-language user that's capacity spent on things he'll never say. V2 is English-only. It's smaller and sharper for his use case. His empirical result is consistent with the architecture.
So the mechanisms are hard and the heuristics fail. The obvious next question is, does any of this actually run on a phone? And how would you know?
The Microsoft report is the most useful thing here, because it does exactly what Daniel wants to do. It measures a full streaming pipeline on CPU with quantization across a lot of configurations. And the headline is that quantization is remarkably lossless for ASR. Nemotron went from two point four seven gigabytes down to zero point six seven. Seventy-three percent smaller. Word error rate degraded zero point one seven percent absolute.
That's basically free.
The int8 variant at one point two eight gigabytes was essentially lossless. Eight point zero one versus eight point zero three. And all the ONNX variants hit better than six times real-time on CPU alone. Zero point five six seconds algorithmic delay, effective time-to-first-token under zero point seven seconds. And here's the part that surprised me. Quantization actually sped up CPU inference. Seven point one five to seven point three zero times, versus six point seven three for full precision.
Smaller and faster. That never happens.
It happens on CPU, where memory bandwidth is the bottleneck. Fewer bytes to move means faster arithmetic. But there's a negative result in that same report that is the strongest possible argument for Daniel's whole premise. The ConvInteger variant, MatMulInteger, degraded to ten point one four percent word error rate. Despite identical int4 weights.
Identical weights, worse result. How?
Integer arithmetic accumulates rounding errors through twenty-four conformer layers. Same numbers going in, different arithmetic, and the error compounds. So the compression scheme that looks equivalent on paper is not equivalent in practice. You cannot predict this from the weight file. You have to run it.
Which is Daniel's entire point. Data beats hunch.
And then the caveat that validates his specific worry about the phone being unusually idle. Those benchmarks ran on an AMD EPYC server CPU pinned to thirty-two cores. The paper itself warns that on weaker consumer hardware the margin shrinks. Under an illustrative assumption of roughly two times lower throughput, the zero point one six second configuration would run at about one point three times real-time. One point three.
So you've gone from six times headroom to almost none.
And one point three times real-time means the moment you open a browser tab, the pipeline stutters. That's the exact scenario Daniel described. The dictation feels wonderful because the phone is idle. Start doing things and the headroom evaporates.
Six times real-time is a server number. It is not a phone expectation.
It is not. And the Parakeet numbers specifically are instructive, because Parakeet is not natively streaming. Its batch word error rate is six point three two percent, which is competitive. But its best chunked configuration was nine point two two percent. That's a forty-six percent relative degradation.
It falls apart when you force it into real time.
Its Batch-to-Stream Factor was two point zero three, versus Nemotron's one point zero three. A BSF of one means no loss from streaming. Parakeet loses half its accuracy. And on CPU it ran at one point three eight times real-time versus Nemotron's two point four six.
So the model Daniel likes is the one that degrades worst under streaming. That's a real tension.
It's a tension, but it's not a contradiction. He's using it for dictation, which is bursty. You speak a sentence, you pause, it processes. He's not running a live captioning stream. The chunking penalty bites hardest when the audio never stops.
Which means Daniel's use case is the one where Parakeet's weakness doesn't matter much. Lucky again.
Now, the benchmarking vocabulary, because this is what he should actually adopt. RTFx, real-time factor, is audio duration divided by wall-clock processing time. It has to stay above one point zero per chunk or you get audio dropout. Effective latency is roughly algorithmic delay times one plus one over RTFx. That formula separates the chunking delay from the compute time, which is how you attribute where the time goes in a chain.
BSF, the batch-to-stream factor, is streaming word error rate over batch word error rate. That's how much accuracy you pay for real time.
One point zero means free. Parakeet's two point zero three means expensive.
On the concurrency question Daniel raised, can you actually run VAD and ASR at the same time, what does the literature say?
It says mostly no, and that's the honest answer. The pipelines are sequential. VAD segments, ASR processes chunks, cleanup runs on the assembled transcript. The closest thing to a concurrent design is WhisperPipe's dynamic buffering with overlapping context windows. It gets eighty-nine milliseconds median end-to-end latency, forty-eight percent less peak GPU memory, and stable memory over a hundred and fifty minutes of continuous operation. Zero growth rate.
Zero growth over two and a half hours. That's the metric a hunch would never catch.
Never. You'd feel it working for ten minutes and call it solved. Two hours in, if memory is creeping, you've got a problem you didn't know you had.
The architecture Daniel's describing, small ASR plus small cleanup model, is already shipping. That's not speculative.
Muesli, from May. Runs on-device on Apple's Neural Engine through CoreML. Uses Parakeet, Whisper, and Cohere transcribe for speech to text, and when you switch on post-processing, the ASR output goes through a fine-tuned Qwen three point five model. That is exactly the chain Daniel wants to benchmark. It exists. It's in production.
The developer's note is interesting. On-device is faster than cloud but a little less accurate, and the cleanup model is what closes the accuracy gap.
Which reframes the cleanup stage entirely. It's not polish. It's the part that makes on-device competitive with cloud.
Then there's the other end of the spectrum. Yap, from July. Ships no model at all.
None. It uses Apple's SpeechAnalyzer and SpeechTranscriber APIs in macOS twenty-six. About three thousand lines of Swift in a four megabyte app, idling near sixty megabytes of memory. And a benchmark in the thread put Apple's on-device model slightly ahead of Whisper Small on accuracy and about three times faster.
The OS vendor absorbed the whole pipeline.
Absorbed it and gave it away. That's the frontier-becomes-widely-available dynamic Daniel described. What's hard today is a system call tomorrow.
The gap Daniel would be filling. I want to be precise about this, because it's the most interesting thing in the research. There is no Android-specific benchmarking literature. None that surfaced. The on-device work is dominated by Apple Neural Engine and generic CPU and ONNX targets.
Which means measurements on a OnePlus Nord 3 would be novel data. Not a blog post repeating what someone already did on a Mac. First data point.
That's a real contribution, and it's sitting in his pocket.
Hilbert: Can I ask you something.
Sure.
Hilbert: When you say cleanup. What do you mean by cleanup. Because I did that for a while and I don't think we meant the same thing.
Go on.
Hilbert: It was a captioning shop. Not the engineering side. The other end. You'd get the raw output from whatever the recognizer produced and you'd fix it by hand before it went out. Eight hours of that and you learn what actually breaks. And it wasn't the punctuation. It wasn't the paragraphs. Those were annoying, but they were honest. You could see them.
So what broke?
Hilbert: The model writing down what somebody meant instead of what they said. You'd have the audio, and the transcript would be cleaner than the audio. Smoother. And you couldn't tell where the machine had tidied up. That's the one that got you in trouble, because downstream nobody knew. The client reads it, it reads fine, and the one sentence that mattered is not what the man said.
You think the cleanup models reproduce that.
Hilbert: They're built to. A model that infers your intent is a model that can be wrong about your intent. That's not a bug you fix, that's the feature working. You asked it to guess and it guessed.
There's a rule in that, isn't there. In what you just described.
Hilbert: There was. We weren't allowed to clean up a file we'd transcribed. Same person, both jobs, no. Because you start hearing what you expect to hear. You read the raw output and your brain fills in the word that should be there and you never check the audio. Different person, cold eyes, and they catch it.
That's the causal supervision principle. That's what that is. Every decision has to be computable from the input up to the decision point. If the transcriber and the cleaner are the same pass, the cleaner is using information that wasn't in the input.
It's why the Muesli chain runs the ASR output through a separate Qwen model instead of having Parakeet clean up after itself. Different model, different pass.
Hilbert: It was probably about liability, honestly. Two people on a file, you can point at who did what. I never found out if it worked. I left before they switched to software. I assume it worked about as well as everything else.
There's a version of this where the cleanup stage is the whole product, and the ASR model is just the input.
That version raises a question I don't think anyone's answered. RTFx measures the ASR model cleanly. Audio in, tokens out, a fixed computation. But if the cleanup stage is where the semantic inference lives, and semantic inference isn't a fixed computation, what does it even mean to measure it? You can time it. You can't tell whether it was right without reading it.
The number tells you it ran. It doesn't tell you it understood.
The OS vendors are absorbing the pipeline anyway. Yap ships no model at all and gets competitive accuracy. So the interesting question isn't whether on-device pipelines work. It's which layer of the stack ends up owning the cleanup stage. If the OS owns it, the whole small-model ecosystem becomes a set of components nobody ships directly.
Which is a strange ending for something Daniel's this excited about. But it's not a sad one, because his measurements would be the first Android data point in a literature dominated by Apple silicon and server CPUs. That's an open contribution. Not a solved problem.
The one thing I'd take from this. The pipeline is a chain, and the weak link isn't the model everyone's benchmarking. It's the cleanup stage, where the semantic work lives, and that's the stage nobody has a good way to measure.
Because a number tells you it ran. It doesn't tell you it understood.
That's the one.
Thanks to Hilbert Flumingtop for producing. This has been My Weird Prompts, the human-AI collaboration podcast. If you want to send us a prompt, email us at show at my weird prompts dot com. We'll be back soon.
See you tomorrow.
