# Four Small Models, One Android Phone: Does It Actually Work? My Weird Prompts — an AI-generated podcast. Daniel Rosehill sends in a prompt; AI hosts Corn and Herman discuss it. Verify important claims independently. - Episode: 5460 - Published: 2026-09-22 - Duration: 22:03 - Audio: https://episodes.myweirdprompts.com/audio/on-device-asr-phone-benchmark.m4a - Tags: speech-recognition, local-ai, ai-inference, gpus, ai-evaluation - Web page: https://myweirdprompts.com/episode/on-device-asr-phone-benchmark/ - JSON: https://myweirdprompts.com/api/episodes/on-device-asr-phone-benchmark.json ## Summary Daniel has been replacing typing with voice for two years, and moving from Whisper to Parakeet V2 on Android was a massive step up. But when your dictation pipeline chains a voice activity detector, an ASR model, and one or more cleanup models, how do you know it's actually working — or is the phone just unusually idle? This episode walks the full chain stage by stage: why ASR models are deliberately partial products, why paragraphing is the great unloved task that even punctuation models skip, and why silence timeouts actively mislead turn detection. Along the way: Parakeet V2 beating V3, quantization tolerance that varies wildly by model, and what happens when a benchmark's six-times-real-time headroom shrinks to almost nothing on weaker hardware. ## The prompt > We've talked about small language models in recent episodes.And I have to be honest, this has got me pretty excited.I'm using Parakeet for on-device speech to text on Android and the result is a massive step up versus Whisper.For anyone curious, I'm using Parakeet V2 specifically, which I found to have better results than V3 using the 0.6 billion parameter model and its five point quantization accuracy the way it's downloadable from Hugging Face. As somebody who has been using voice to text almost exclusively for almost two years now, I've spent a considerable amount of effort trying to find the best solutions for dictation with the idea that it will fully replace typing.ASR models are great and it's probably fair to say that it's the most active space of development in voice technology, although there's also some multimodal models.The downside of ASR is that although this varies by model, it's often a partial product that the model will produce whether you're using it locally or via the cloud.We're finally beginning to see the beginning of a mature set of small models specifically for post-transcription cleanup, which is actually a variety of tasks, including punctuation restoration. It could include ignoring inferred comments, like if you were dictating and then said something addressing somebody else in the room, which requires some inference about speaker intention.Another accessory model is diorization. And as I've often mentioned, paragraphs are the great unloved part of it that evades attention of even punctuation or steroid models, but I think it's essential for producing coherent text. And I don't think that red jacks or non-semantic heuristics really cut it.I very much agree with the linguist you mentioned who argued that how one adds paragraphs is an important stylistic feature of their use of language and bounds their thoughts. As I've been experimenting with some of these local apps on Android, checking out the tools, I noticed that we're gradually getting to a point in which the cleanup models are being quantized and released for on-device inference.This means that we're looking at a ASR pipeline now that's being run on devices.For what is worth I have a one plus Nord 3.Not a flagship phone.And I always try out these small models expecting that they either won't work or the inference will become so slow that it's unusable.But I'm pleasantly surprised to see that some of these pipelines are actually viable, and if that's the case what's Frontier today will be widely available tomorrow.I feel it would be useful to run a few benchmarks, ideally on Android and also maybe on my usual computing device and my laptop to measure inference and resource usage across a chained pipeline like an ASR pipeline.Because it's one thing to feel that the inference is running smoothly and that the you finally found a quantization that doesn't challenge the hardware.But it's easy to miss things.Maybe the phone is just unusually idle and the dictation pipeline you think is wonderful now will struggle the moment you start doing things on your phone. The other constituent element of ASR pipelines that I didn't mention is VAD.I benchmarked models in isolation.But I haven't done any measurements or benchmarking of a scenario which is becoming increasingly common and feasible in which multiple small models or maybe a small model and a large model are all running locally in parallel or in close succession. I don't know if it's possible even to have VAD and an ASR model both running concurrent inference before passing to a small model, another small model and then producing text.But if it is, it would be highly useful to see even one or two traces of that, looking at the total RAM consumption at every stage of one dictation, and perhaps being able to visualize the resource usage imposed by each constituent element.I'd imagine one might be pleasantly surprised and find discover that the small models are not taxing the hardware, but even if that were the case having that data would be very useful than going by hunch.Let's look today at the question of running multiple small models and inference chains on local hardware and how to gain some data on what's actually working so that you can match the models you need to the hardware they're going to be running on. ## Show notes The question is simple and hard to answer: if you're running four small models in a chain on a mid-range Android phone and everything feels smooth, how do you know it's working — or is the phone just unusually idle? A listener who's been using voice-to-text almost exclusively for two years wants to benchmark exactly that. He moved from Whisper to Parakeet V2 for on-device ASR, running the 0.6B parameter model at five-point quantization, and found V2 beat V3. He also wants to measure RAM at every stage of a single dictation, on a OnePlus Nord 3, a desktop, and a laptop, because feeling that inference is smooth is not the same as knowing it. The chain has at least three stages. Voice activity detection is the bouncer, deciding what's speech and what's refrigerator noise. The ASR model produces the raw transcript. Cleanup — one or more models — turns that into something a human would read. The ASR model is a partial product by design, not by failure: a Microsoft on-device streaming ASR report from April states outright that its evaluation covers neither inverse text normalization, nor diarization, nor code-switching or custom vocabulary. The cleanup taxonomy is a bundle of different problems wearing similar clothes: punctuation restoration, disfluency removal, inverse text normalization, diarization, turn detection, and paragraphing. The asymmetry organizing the whole discussion is that punctuation restoration is well-studied while paragraphing is essentially absent from the literature. Searches for a dedicated paper on paragraph segmentation for dictation come back empty. And paragraphing is a stylistic feature — how you break paragraphs is part of how you write, not decoration on top of the content. VAD looks trivial and isn't, because the choice of VAD changes what the ASR model even sees. A streaming architecture combining Silero VAD with energy-based filtering cut false activations by thirty-four percent — a quality improvement, not just a latency one. Fewer false activations means fewer phantom segments for the ASR model to hallucinate words into. Turn detection is harder still. The obvious heuristic is silence, and it fails: within-turn pauses routinely exceed between-turn gaps, so a silence timeout cannot separate them. What distinguishes them is whether the words so far form a complete thought. One fix trains a small LoRA adapter on Qwen3-ASR-0.6B for hours on a single GPU, transcribing and detecting end-of-turn from meaning, reaching ninety-seven percent boundary recall at a median 0.39 seconds latency with 0.3 false fires per speech-minute. A principle from that work generalizes: every streaming-decision label has to be computable from the input up to the decision point, or the system oscillates. The empirical findings connect to mechanism. Quantization tolerance is model-specific — one model's int4 variant barely moves while Parakeet's degrades substantially under the same scheme. V2 beating V3 is consistent with architecture: V3 is multilingual and spreads capacity across languages a single-language user will never speak. The Microsoft report shows quantization can be remarkably lossless for ASR, shrinking a model by seventy-three percent for 0.17 percent absolute word error rate degradation, and even speeding up CPU inference because memory bandwidth is the bottleneck. But a ConvInteger variant with identical int4 weights degraded badly, because integer arithmetic accumulates rounding errors through twenty-four conformer layers. You cannot predict this from the weight file — you have to run it. And the benchmarks ran on a 32-core server CPU; the paper warns the margin shrinks on weaker hardware, where a configuration could drop to about 1.3 times real-time. That's the exact scenario in question: dictation feels wonderful because the phone is idle, and the headroom evaporates the moment you open a browser tab. ## Transcript Corn: Here's the question that's been rattling around my head since Daniel sent this in. If you've got four small models running in a chain on a mid-range Android phone, and everything feels smooth, how do you actually know it's working? Or is the phone just unusually idle? Herman: That's the whole episode right there. Corn: It is. Daniel's been using voice-to-text almost exclusively for about two years now, with the goal of fully replacing typing. And he's excited, excited, because he moved from Whisper to Parakeet V2 for on-device speech to text on Android and says it's a massive step up. He's using the 0.6 billion parameter model, five-point quantization, downloaded from Hugging Face. And he found V2 beat V3, which is its own little mystery we'll get into. Herman: The English-only thing. Corn: Right. But here's the part he actually wants to talk about. He says ASR is probably the most active space in voice technology right now, and the models are great, but they're often a partial product. They give you a raw transcript and stop. So we're finally seeing a mature set of small models specifically for post-transcription cleanup, which is really a bundle of different tasks. Punctuation restoration. Ignoring inferred comments, which means if you're dictating and then you say something to somebody else in the room, the model has to infer that wasn't meant for the transcript. Diarization. And paragraphing, which Daniel calls the great unloved part, the thing that evades the attention of even punctuation models, and which he thinks non-semantic heuristics can't solve. Herman: He's right about that. Corn: He wants to benchmark a chained pipeline. VAD, ASR, one or more cleanup models, ideally on his OnePlus Nord 3, which he's careful to note is not a flagship, and also on his usual machine and his laptop. Measuring RAM at every stage of a single dictation. Because feeling that inference is smooth is not the same as knowing it. The phone might just be idle. And he wants to know whether you can even run VAD and an ASR model concurrently before passing to a cleanup model, and then another one, and whether anyone has traced the resource usage of each constituent element. Herman: So let's start with what's actually in the chain, because it's more than one model. Corn: Go. Herman: The way to think about it is at least three stages. Stage one is voice activity detection. It's the bouncer. It decides what's speech and what's silence or noise, so the ASR model isn't burning compute on the sound of your refrigerator. Stage two is the ASR model itself, which produces the raw transcript. Stage three is the cleanup, which is one or more models turning that raw transcript into something a human would actually want to read. And the crucial thing, the thing Daniel's pointing at, is that the ASR model is a partial product by design, not by failure. Herman: It's a concession from the people building it. Microsoft's CoreAI group put out an on-device streaming ASR report in April, and they say it outright. Their evaluation doesn't cover inverse text normalization, which is numbers, dates, currencies. It doesn't cover speaker diarization. It doesn't cover code-switching or custom vocabulary adaptation. That's a frontier lab saying, here's our model, and here's the list of things it doesn't do. Corn: So the model that everybody's benchmarking is one component of a larger system that nobody's benchmarking. Herman: And the cleanup taxonomy is worth naming concretely, because Daniel lists them and they're different problems wearing similar clothes. Punctuation restoration. Disfluency removal, so the ums and the false starts. Inverse text normalization, which is turning "twenty three dollars and fifty cents" into the numeral form. Diarization, who said what. Turn detection, which is where the ignoring-inferred-comments thing lives. And paragraphing. Corn: And the asymmetry that organizes this whole episode is that punctuation restoration is well-studied, and paragraphing is basically absent from the literature. Daniel says it's unloved. I went looking for a dedicated paper on paragraph segmentation for dictation and found nothing. Searches came back empty. That's not a gap in my search technique, that's a gap in the field. Herman: The silence is the evidence. Corn: The map is clear enough. The question is why the cleanup stage is so much harder than it looks, and that starts with VAD. Herman: VAD looks like a pre-filter. It looks like the trivial part. And it isn't, because the choice of VAD changes what the ASR model even sees. WhisperPipe, which is a streaming architecture paper from April, combines Silero VAD with energy-based filtering, and that combination cut false activations by thirty-four percent. That's not a latency improvement, that's a quality improvement. Fewer false activations means fewer phantom segments for the ASR model to hallucinate words into. Corn: So the bouncer isn't just deciding who gets in. The bouncer is deciding what the band plays. Herman: That's the shape of it, yes. And it gets worse, because the real problem is turn detection. This is the ignoring-inferred-comments task Daniel described. If you're dictating and you turn to your kid and say "put your shoes on," the pipeline needs to know that wasn't part of the document. And the obvious heuristic is silence. A long pause means the turn ended, right? Corn: It does not mean that. Herman: It does not. There's a paper from July, "The Trade-off Was in the Labels," that shows within-turn pauses routinely exceed between-turn gaps. So if you're a VAD with a silence timeout, you cannot separate them. The pause inside a thought is often longer than the pause between two thoughts. What distinguishes them is whether the words so far form a complete thought. That's it. That's the only signal that works. Corn: So the acoustic information is actively misleading. Herman: Actively. And their fix is elegant. They train a small LoRA adapter on Qwen3-ASR-0.6B, hours on one GPU, and it transcribes and detects end-of-turn from meaning. Ninety-seven percent boundary recall at a median of zero point three nine seconds latency, with zero point three false fires per speech-minute. Corn: Point three nine seconds. That's the number that matters, because that's the delay between you finishing a thought and the system knowing you finished it. Herman: And paragraphing is the same problem in a different costume. Daniel's right that non-semantic heuristics don't cut it. Pause length, token count, time gaps. They all fail for exactly the reason the silence timeout fails at turn detection. They can't see semantic completeness. A paragraph bounds a thought. If your heuristic can't see thoughts, it can't see paragraphs. Corn: And I'll say the thing Daniel said, because he's right and it deserves saying plainly. Paragraphing is a stylistic feature. How you break your paragraphs is part of how you write. It's not decoration on top of the content, it's part of the content. He agreed with the linguist we had on who argued exactly that, that paragraphing bounds your thoughts. And a model that's just counting tokens is not doing style, it's doing layout. Herman: There's a principle in the turn-detection paper that I think is the technical version of what you just said. Every streaming-decision label has to be computable from the input up to the decision point. That's the causal supervision principle. If your label depends on information you don't have yet at the moment you have to make the decision, you're going to oscillate. You'll put a paragraph break here, then the next word arrives and now the break was wrong. Corn: So the naive heuristic doesn't just fail, it fails noisily. It flip-flops. Herman: Right. And FastTurn, another paper from this year, is the honest framing of the whole design space. It unifies acoustic and streaming semantic cues for turn detection, and it says the quiet part out loud. Pure VAD approaches lack semantic understanding. Pure ASR approaches introduce latency. You're not looking for the right answer, you're picking a point on a tradeoff curve. Corn: Which brings us back to Daniel's actual empirical finding, which I want to sit with, because it's not a fluke. He found Parakeet V2 beat V3. Same family, newer version, worse result for him. Herman: And the quantization data explains why. Quantization tolerance is model-specific. Nemotron's int4 variant barely moves, eight point zero three percent word error rate down to eight point two zero. Parakeet's int4 goes to eleven point five zero percent. Same quantization scheme, wildly different damage. Corn: So when Daniel says he found the five-point quantization that doesn't challenge his hardware, he's describing a property of Parakeet specifically. He got lucky in the sense that his model tolerates what he did to it. A different model at the same compression would have fallen apart. Herman: And V2 versus V3 fits the same story. V3 is multilingual, spreads its capacity across languages, and for a single-language user that's capacity spent on things he'll never say. V2 is English-only. It's smaller and sharper for his use case. His empirical result is consistent with the architecture. Corn: So the mechanisms are hard and the heuristics fail. The obvious next question is, does any of this actually run on a phone? And how would you know? Herman: The Microsoft report is the most useful thing here, because it does exactly what Daniel wants to do. It measures a full streaming pipeline on CPU with quantization across a lot of configurations. And the headline is that quantization is remarkably lossless for ASR. Nemotron went from two point four seven gigabytes down to zero point six seven. Seventy-three percent smaller. Word error rate degraded zero point one seven percent absolute. Corn: That's basically free. Herman: The int8 variant at one point two eight gigabytes was essentially lossless. Eight point zero one versus eight point zero three. And all the ONNX variants hit better than six times real-time on CPU alone. Zero point five six seconds algorithmic delay, effective time-to-first-token under zero point seven seconds. And here's the part that surprised me. Quantization actually sped up CPU inference. Seven point one five to seven point three zero times, versus six point seven three for full precision. Corn: Smaller and faster. That never happens. Herman: It happens on CPU, where memory bandwidth is the bottleneck. Fewer bytes to move means faster arithmetic. But there's a negative result in that same report that is the strongest possible argument for Daniel's whole premise. The ConvInteger variant, MatMulInteger, degraded to ten point one four percent word error rate. Despite identical int4 weights. Corn: Identical weights, worse result. How? Herman: Integer arithmetic accumulates rounding errors through twenty-four conformer layers. Same numbers going in, different arithmetic, and the error compounds. So the compression scheme that looks equivalent on paper is not equivalent in practice. You cannot predict this from the weight file. You have to run it. Corn: Which is Daniel's entire point. Data beats hunch. Herman: And then the caveat that validates his specific worry about the phone being unusually idle. Those benchmarks ran on an AMD EPYC server CPU pinned to thirty-two cores. The paper itself warns that on weaker consumer hardware the margin shrinks. Under an illustrative assumption of roughly two times lower throughput, the zero point one six second configuration would run at about one point three times real-time. One point three. Corn: So you've gone from six times headroom to almost none. Herman: And one point three times real-time means the moment you open a browser tab, the pipeline stutters. That's the exact scenario Daniel described. The dictation feels wonderful because the phone is idle. Start doing things and the headroom evaporates. Corn: Six times real-time is a server number. It is not a phone expectation. Herman: It is not. And the Parakeet numbers specifically are instructive, because Parakeet is not natively streaming. Its batch word error rate is six point three two percent, which is competitive. But its best chunked configuration was nine point two two percent. That's a forty-six percent relative degradation. Corn: It falls apart when you force it into real time. Herman: Its Batch-to-Stream Factor was two point zero three, versus Nemotron's one point zero three. A BSF of one means no loss from streaming. Parakeet loses half its accuracy. And on CPU it ran at one point three eight times real-time versus Nemotron's two point four six. Corn: So the model Daniel likes is the one that degrades worst under streaming. That's a real tension. Herman: It's a tension, but it's not a contradiction. He's using it for dictation, which is bursty. You speak a sentence, you pause, it processes. He's not running a live captioning stream. The chunking penalty bites hardest when the audio never stops. Corn: Which means Daniel's use case is the one where Parakeet's weakness doesn't matter much. Lucky again. Herman: Now, the benchmarking vocabulary, because this is what he should actually adopt. RTFx, real-time factor, is audio duration divided by wall-clock processing time. It has to stay above one point zero per chunk or you get audio dropout. Effective latency is roughly algorithmic delay times one plus one over RTFx. That formula separates the chunking delay from the compute time, which is how you attribute where the time goes in a chain. Corn: BSF, the batch-to-stream factor, is streaming word error rate over batch word error rate. That's how much accuracy you pay for real time. Herman: One point zero means free. Parakeet's two point zero three means expensive. Corn: On the concurrency question Daniel raised, can you actually run VAD and ASR at the same time, what does the literature say? Herman: It says mostly no, and that's the honest answer. The pipelines are sequential. VAD segments, ASR processes chunks, cleanup runs on the assembled transcript. The closest thing to a concurrent design is WhisperPipe's dynamic buffering with overlapping context windows. It gets eighty-nine milliseconds median end-to-end latency, forty-eight percent less peak GPU memory, and stable memory over a hundred and fifty minutes of continuous operation. Zero growth rate. Corn: Zero growth over two and a half hours. That's the metric a hunch would never catch. Herman: Never. You'd feel it working for ten minutes and call it solved. Two hours in, if memory is creeping, you've got a problem you didn't know you had. Corn: The architecture Daniel's describing, small ASR plus small cleanup model, is already shipping. That's not speculative. Herman: Muesli, from May. Runs on-device on Apple's Neural Engine through CoreML. Uses Parakeet, Whisper, and Cohere transcribe for speech to text, and when you switch on post-processing, the ASR output goes through a fine-tuned Qwen three point five model. That is exactly the chain Daniel wants to benchmark. It exists. It's in production. Corn: The developer's note is interesting. On-device is faster than cloud but a little less accurate, and the cleanup model is what closes the accuracy gap. Herman: Which reframes the cleanup stage entirely. It's not polish. It's the part that makes on-device competitive with cloud. Corn: Then there's the other end of the spectrum. Yap, from July. Ships no model at all. Herman: None. It uses Apple's SpeechAnalyzer and SpeechTranscriber APIs in macOS twenty-six. About three thousand lines of Swift in a four megabyte app, idling near sixty megabytes of memory. And a benchmark in the thread put Apple's on-device model slightly ahead of Whisper Small on accuracy and about three times faster. Corn: The OS vendor absorbed the whole pipeline. Herman: Absorbed it and gave it away. That's the frontier-becomes-widely-available dynamic Daniel described. What's hard today is a system call tomorrow. Corn: The gap Daniel would be filling. I want to be precise about this, because it's the most interesting thing in the research. There is no Android-specific benchmarking literature. None that surfaced. The on-device work is dominated by Apple Neural Engine and generic CPU and ONNX targets. Herman: Which means measurements on a OnePlus Nord 3 would be novel data. Not a blog post repeating what someone already did on a Mac. First data point. Corn: That's a real contribution, and it's sitting in his pocket. Hilbert: Can I ask you something. Herman: Sure. Hilbert: When you say cleanup. What do you mean by cleanup. Because I did that for a while and I don't think we meant the same thing. Corn: Go on. Hilbert: It was a captioning shop. Not the engineering side. The other end. You'd get the raw output from whatever the recognizer produced and you'd fix it by hand before it went out. Eight hours of that and you learn what actually breaks. And it wasn't the punctuation. It wasn't the paragraphs. Those were annoying, but they were honest. You could see them. Corn: So what broke? Hilbert: The model writing down what somebody meant instead of what they said. You'd have the audio, and the transcript would be cleaner than the audio. Smoother. And you couldn't tell where the machine had tidied up. That's the one that got you in trouble, because downstream nobody knew. The client reads it, it reads fine, and the one sentence that mattered is not what the man said. Herman: You think the cleanup models reproduce that. Hilbert: They're built to. A model that infers your intent is a model that can be wrong about your intent. That's not a bug you fix, that's the feature working. You asked it to guess and it guessed. Corn: There's a rule in that, isn't there. In what you just described. Hilbert: There was. We weren't allowed to clean up a file we'd transcribed. Same person, both jobs, no. Because you start hearing what you expect to hear. You read the raw output and your brain fills in the word that should be there and you never check the audio. Different person, cold eyes, and they catch it. Herman: That's the causal supervision principle. That's what that is. Every decision has to be computable from the input up to the decision point. If the transcriber and the cleaner are the same pass, the cleaner is using information that wasn't in the input. Corn: It's why the Muesli chain runs the ASR output through a separate Qwen model instead of having Parakeet clean up after itself. Different model, different pass. Hilbert: It was probably about liability, honestly. Two people on a file, you can point at who did what. I never found out if it worked. I left before they switched to software. I assume it worked about as well as everything else. Corn: There's a version of this where the cleanup stage is the whole product, and the ASR model is just the input. Herman: That version raises a question I don't think anyone's answered. RTFx measures the ASR model cleanly. Audio in, tokens out, a fixed computation. But if the cleanup stage is where the semantic inference lives, and semantic inference isn't a fixed computation, what does it even mean to measure it? You can time it. You can't tell whether it was right without reading it. Corn: The number tells you it ran. It doesn't tell you it understood. Herman: The OS vendors are absorbing the pipeline anyway. Yap ships no model at all and gets competitive accuracy. So the interesting question isn't whether on-device pipelines work. It's which layer of the stack ends up owning the cleanup stage. If the OS owns it, the whole small-model ecosystem becomes a set of components nobody ships directly. Corn: Which is a strange ending for something Daniel's this excited about. But it's not a sad one, because his measurements would be the first Android data point in a literature dominated by Apple silicon and server CPUs. That's an open contribution. Not a solved problem. Herman: The one thing I'd take from this. The pipeline is a chain, and the weak link isn't the model everyone's benchmarking. It's the cleanup stage, where the semantic work lives, and that's the stage nobody has a good way to measure. Corn: Because a number tells you it ran. It doesn't tell you it understood. Herman: That's the one. Corn: Thanks to Hilbert Flumingtop for producing. This has been My Weird Prompts, the human-AI collaboration podcast. If you want to send us a prompt, email us at show at my weird prompts dot com. We'll be back soon. Herman: See you tomorrow.