Daniel's been digging into on-device transcription again, and this time he's gone below the model layer entirely. He wants to talk runtimes, compatibility, what a TPU actually changes, and how to find an ASR model on Hugging Face that won't just crash your phone. He's used to sizing desktop models by VRAM, and he's noticed Android doesn't give you that same single number. He also mentions his wife picked up a Pixel 11 and has been downloading TPU-optimized variants from AI Edge Gallery, which he recommends as a quick way to test whether your device can run local models at all. Then he wants the emerging options beyond Moonshine and Parakeet, and what they're actually optimizing for.
So let's start with the layer most people never see. The runtimes that sit between the model file and the silicon.
Right. Because when you download a model from Hugging Face, you're not downloading something that runs. You're downloading weights. A graph of operations. Something has to load that file, map every operator onto whatever hardware is available, and manage the memory while it's doing it. That something is the runtime.
And Android has a fragmented runtime landscape. You've got LiteRT, which is Google's successor to TensorFlow Lite, and it's the runtime behind AI Edge Gallery. It has an NPU delegate that lets models run on the Tensor chip's TPU. Then there's ONNX Runtime Mobile, which is cross-platform and has a broad operator set, but on Android it frequently falls back to CPU unless there's a vendor-specific execution provider available. Then ExecuTorch, which is PyTorch's on-device runtime, still maturing but gaining traction because researchers can ship models without a separate conversion pipeline. And llama.cpp, which is technically an LLM runtime, but it's increasingly used for ASR models converted to GGUF, including Whisper variants.
And then the vendor stacks underneath all of that. Qualcomm's QNN, MediaTek's NeuroPilot. The things nobody wants to think about until their model loads and immediately dies.
The runtime choice often determines which models you can even attempt. A model compiled for one runtime's operator set may not load in another at all. It's not like desktop where you mostly pick PyTorch or ONNX and move on. On Android, the runtime is the first compatibility gate.
So the question Daniel's really asking is, why is there no VRAM equivalent on a phone? And the answer is that on desktop, VRAM is a single pool with a single number. You know if a model fits. On Android, you've got system RAM, but you've also got thermal headroom, memory bandwidth, NPU availability, OS-level process limits, and the runtime's own overhead. Any of those can be the thing that fails first.
And they fail differently. A model can fit in RAM and still throttle to uselessness because it's drawing too much power for sustained transcription. Or it can fit and run, but the runtime's memory allocator fragments after twenty minutes and the whole thing falls over. Desktop VRAM never does that.
So the single number was always a convenience, not a law of physics. Phones just refuse to give you the convenience.
Let's talk about LiteRT first, because it's the one most people will actually encounter. It's Google's successor to TensorFlow Lite, and the NPU delegate is the key piece. When you load a model through LiteRT with the NPU delegate enabled, the runtime partitions the graph. Operators the TPU supports get sent to the TPU. Everything else stays on CPU. The delegate is the handshake that makes that happen.
And AI Edge Gallery is built on top of this. It's Google's sandbox for trying on-device AI, and it ships TPU-optimized variants of models for Pixel devices. So when Daniel's wife downloads a TPU-optimized model in the gallery, she's getting a file that's been converted and compiled specifically for the Tensor TPU, often with int8 or int4 quantization and operator fusion tuned for that silicon.
What that means in practice is that the graph has been partitioned ahead of time. Operators are mapped to TPU-supported equivalents. The result is a tflite file with the TPU delegate baked in, so most of the compute runs on the TPU instead of the CPU. Lower latency, lower power draw. For transcription, which is a sustained workload, that's the difference between a phone that gets warm and a phone that gets hot.
ONNX Runtime Mobile is the other big one. Its strength is portability. You can take the same ONNX file and run it on Windows, Linux, iOS, Android. Broad operator set. But on Android, unless you have a vendor execution provider, it falls back to CPU. And an execution provider is just the backend that maps operators to specific hardware. Qualcomm has one for their NPU. MediaTek has one. If you don't have the right one for your chip, you're on CPU, and CPU transcription on a phone is slow.
ExecuTorch is interesting because it's PyTorch's answer to this whole mess. The pitch is that you take a model you trained in PyTorch, export it once, and run it on device without a separate conversion pipeline. That's appealing for researchers who don't want to learn TFLite's quirks. The Android support is through Java and Kotlin bindings. It's still maturing, but it's gaining traction because the friction is lower.
And llama.cpp is the odd one out. It's an LLM runtime, but the GGUF format has become a kind of universal container for quantized models, including Whisper variants. The advantage is it runs almost anywhere. The disadvantage is it doesn't use the NPU or TPU at all. It's CPU-only, maybe GPU if you're on desktop. On a phone, that means it works, but you're leaving the most efficient silicon on the device completely idle.
The operator compatibility problem is where all of this gets concrete. A model is a graph of operators. Convolutions, attention, layer norm, that kind of thing. Each runtime implements a subset. If your model uses an operator the runtime doesn't support, one of two things happens. Either the model fails to load entirely, or the runtime falls back to CPU for that layer. And a single CPU layer in the middle of a TPU-accelerated graph can tank performance, because now you're shuttling tensors back and forth across the memory bus.
So you can have a model that's ninety-five percent TPU-accelerated, and the five percent that isn't becomes the bottleneck. It's like having a highway with one traffic light in the middle.
And this is why parameter count is such a poor sizing proxy on Android. On desktop, more parameters means more VRAM, full stop. On Android, a hundred million parameter model in float32 is roughly four hundred megabytes. The same model in int8 is roughly a hundred megabytes. Same parameter count, quarter of the memory. Quantization changes the footprint dramatically. Operator fusion changes it again. And the NPU has its own memory constraints that are completely separate from system RAM.
So you could have a model that fits comfortably in system RAM and still not fit on the TPU's local memory. Or the reverse. The parameter count tells you almost nothing about either.
Whisper Tiny is the classic example. Thirty-nine million parameters. If you run it through LiteRT with int8 quantization, you're looking at roughly forty megabytes. If you run the same model through ONNX Runtime in float32, it's over a hundred and fifty megabytes. Same model. Same parameter count. And then the question of whether the encoder and decoder are split across CPU and NPU changes the latency profile entirely.
So when someone says "my phone can run a hundred million parameter model," the correct response is, which runtime, which quantization, which operators, and what's the thermal budget?
And that last one matters more than people think. Transcription isn't a benchmark. It's a sustained workload. You're not running one inference, you're running hundreds of them back to back as audio streams in. A phone that can spike to peak performance for ten seconds will throttle to half that after two minutes of continuous transcription. The TPU helps because it's more efficient per operation, but the thermal envelope is still the ceiling.
So that's the runtime layer. Now let's talk about what changes when you add a TPU to the mix, because Daniel's wife's Pixel 11 is a different deployment target than a phone without one.
The Pixel 11 uses Google's Tensor chip, which includes a dedicated TPU. And Google ships TPU-optimized model variants in AI Edge Gallery. These are models that have been converted and compiled specifically for the Tensor TPU, often with int8 or int4 quantization and operator fusion tuned for that silicon.
So when we say TPU-optimized, we mean the graph has been partitioned, operators mapped to TPU-supported equivalents, and the result is a tflite file with the TPU delegate that runs the bulk of the compute on the TPU. The payoff is lower latency and lower power draw. For transcription, that's the difference between a phone that can transcribe all day and a phone that gives you twenty minutes before it's too hot to hold.
The tradeoff is portability. A model compiled for the Tensor TPU won't run on a Snapdragon NPU or a MediaTek APU. You're trading flexibility for performance. And this is the thing people miss when they see TPU-optimized and assume it's universally better. It's better on that specific TPU. On other hardware, it may not run at all.
Which is a real consideration if you're building an app. Do you ship one model that runs everywhere but slower, or do you ship three models and detect the hardware at runtime? Most apps do the latter, but it's a build and maintenance burden.
And the fragmentation is only going to get worse as more vendors ship their own NPUs. Qualcomm has its own. MediaTek has its own. Samsung has its own. Google has the TPU. Each one has its own operator set, its own quantization preferences, its own execution provider. The runtime layer is supposed to abstract this, but the abstraction leaks constantly.
Let's get practical. Daniel asked for concrete tips on finding an ASR model on Hugging Face that will actually run on Android. What's the workflow?
First, filter by tags. Hugging Face model cards have tags, and you can filter for LiteRT or TFLite. That immediately narrows you to models that have been converted for mobile. Second, look for an Android or on-device deployment section in the README. If the model author bothered to document mobile deployment, that's a strong signal they've actually tested it. Third, check the quantization info. int8 is the sweet spot for mobile ASR. It's accurate enough for transcription and small enough to fit in NPU memory.
Fourth, look for models that ship with a tflite or onnx file directly, not just PyTorch weights. If you have to convert it yourself, you're signing up for a whole afternoon of operator compatibility debugging. Fifth, check the operator list against the runtime's supported operators. Most model cards won't list this explicitly, but if the model uses something exotic, it'll usually show up in the issues or discussions.
Sixth, prefer models with a parameter count under about two hundred million for sustained ASR on mid-range devices. That's not because the parameter count itself is the limit. It's because models in that range tend to quantize well and fit in NPU memory. Above that, you're fighting the thermal budget even if the model technically loads.
That's the discovery workflow. Filter by tags, read the README, check quantization, prefer pre-converted files, verify operators, and stay under two hundred million parameters unless you have a reason not to.
The reason not to would be something like, you need multilingual coverage and the only model that does it well is bigger. Or you're running on a flagship with a TPU and you can afford it. But for most people, the sweet spot is smaller than they think.
Now the emerging options. Daniel mentioned Moonshine and Parakeet, but the space is moving fast. The newer entrants tend to target specific tradeoffs. Some prioritize streaming, which means low-latency chunked inference. The model starts transcribing before the sentence is finished. That's critical for live captions or dictation. Others prioritize multilingual coverage. Others prioritize tiny footprint for wearables and earbuds.
Many of these ship as LiteRT or ONNX models first, with PyTorch weights as an afterthought. That's a reversal from the desktop world, where PyTorch is the default and everything else is a conversion. On mobile, the deployment format is the product.
The practical implication is that the best ASR model on Android is not the most accurate one. It's the one that fits your runtime, your silicon, and your thermal budget. Accuracy benchmarks on desktop GPUs are misleading for mobile deployment. A model that wins on a desktop benchmark might be unusable on a phone because it uses an operator the NPU doesn't support, or it's too big to quantize well, or it draws too much power for sustained use.
That's the shift in mindset. On desktop, you look at the leaderboard and pick the top model. On Android, you look at the runtime support first, then the quantization, then the accuracy. The model that wins is the one that actually runs.
What's actually out there beyond Moonshine and Parakeet? The streaming models are interesting because they're designed around the chunked inference pattern that mobile transcription needs. They don't wait for the full utterance. They process audio as it arrives and emit tokens incrementally. That's a fundamentally different architecture than batch models, and it changes what the runtime has to do.
The multilingual models are also getting better. Whisper variants with expanded language coverage, some of them distilled down to tiny sizes. The distillation is the interesting part. You train a large model, then use it to train a smaller model that mimics its outputs. You get most of the accuracy at a fraction of the size. That's how you get a forty million parameter model that transcribes twenty languages reasonably well.
Then there's the wearable-tier stuff. Models with ten or twenty million parameters that run on earbuds. Those are int4 quantized, heavily fused, and they do one thing: wake word detection plus short command transcription. Not general ASR, but the same runtime considerations apply.
What I'm watching is the convergence of streaming and multilingual. The first model that does both well at under a hundred million parameters with clean LiteRT support is going to become the default for a lot of apps. Nobody's quite there yet, but the gap is closing.
The runtime layer is the thing that determines what you can even attempt. The TPU changes the math for specific devices but fragments the ecosystem. And the discovery workflow on Hugging Face is a set of filters and signals that most people skip because they're used to desktop, where you just download the biggest model that fits in VRAM.
The thermal budget is the thing that bites you later. You can get a model running on a phone and think you're done, then find out it throttles after ten minutes of continuous transcription. The benchmark looked great. The real-world experience is a phone that's hot and a transcription that's dropping words.
The runtime's memory allocator matters too. Some runtimes fragment memory over long sessions. LiteRT is generally solid here. ONNX Runtime Mobile has improved. ExecuTorch is still working on it. If you're building an app that transcribes for hours, you need to test for memory leaks and fragmentation, not just accuracy.
That's the kind of thing that never shows up in a model card. The model author tested it for five minutes on their desk and called it good. The person who runs it for three hours in the field finds the problem.
The question Daniel's really circling is, how do you make a decision in an environment where the ground keeps shifting? The runtimes are still maturing. The silicon is fragmenting. The models are getting better every month. What's the stable principle?
The stable principle is that the runtime is the first gate. Before you ask whether a model is accurate, ask whether it loads. Before you ask whether it fits, ask which runtime and which quantization. And before you ship, test it for the actual workload: sustained transcription, not a single inference.
The second stable principle is that portability and performance are in tension. The more you optimize for a specific TPU, the less portable you are. The more portable you are, the less you're using the specific silicon. There's no free lunch.
The third is that parameter count is a proxy that mostly works on desktop and mostly doesn't on mobile. The real constraints are quantization, operator support, and thermal headroom. If you hold those three in your head, you can evaluate any model on any device.
When someone says "my phone can run a hundred million parameter model," the correct response is, which runtime, which quantization, which operators, and what's the thermal budget?
That's the thing I keep coming back to. The desktop mindset is a single number. The mobile reality is a set of interacting constraints. Once you internalize that, the rest is just details.
What happens when you add a TPU? You get a new constraint and a new capability at the same time. The TPU can run int8 operations extremely efficiently, which means you can run bigger models at lower power. But you also now have a second memory pool to worry about, and a second operator set to check.
The TPU's memory is separate from system RAM. A model that fits in system RAM might not fit in TPU memory. And the TPU's operator support is narrower than the CPU's. So you have to check both. The TPU delegate handles the partitioning, but you need to know what's actually going to the TPU and what's staying on CPU.
The AI Edge Gallery is the easiest way to see this in action. You download a TPU-optimized model, run it, and watch the latency and power draw. Then download the non-optimized version of the same model and compare. The difference is usually dramatic.
That's one of the reasons Daniel recommends it. It's a quick way to test whether your device can run local models at all, and it gives you a baseline for what's possible. You can then take that baseline to Hugging Face and look for models that match it.
The workflow is: test your device with AI Edge Gallery, see what runs and how fast, then go to Hugging Face with that knowledge and filter for models that match your runtime and quantization. That's a much better starting point than just downloading the most popular ASR model and hoping.
The model card tags are the first filter. LiteRT, TFLite, onnx, gguf. Those tell you what format the model is actually available in. If it only has PyTorch weights, you're signing up for conversion work. If it has a tflite file, someone already did that work for you.
The README is the second filter. A model author who's actually tested on Android will say so. They'll mention which devices they tested on, which runtime they used, what the latency was. That's gold. A model card with no deployment section is a gamble.
The quantization info is the third filter. int8 is the sweet spot. float16 is workable if you have a TPU. float32 is desktop territory. int4 is aggressive but works for tiny models. If the model card doesn't mention quantization, assume float32 and move on.
Then the operator check. This is the one most people skip because it's tedious. But if the model uses an operator your runtime doesn't support, it'll either fail to load or fall back to CPU for that layer. And a CPU layer in the middle of a TPU graph is a performance disaster.
The parameter count is the last filter, not the first. Under two hundred million for sustained ASR on mid-range devices. Above that, you're fighting the thermal budget even if the model loads.
That's the concrete workflow. And the emerging options are all over this map. Some are streaming-first, some are multilingual-first, some are tiny-footprint-first. The one you pick depends on which constraint matters most for your use case.
The use case matters more than people think. Live captions need streaming. Podcast transcription can use batch. Multilingual dictation needs broad language coverage. Voice commands for an app need a tiny model that responds instantly. These are different models, different runtimes, different tradeoffs.
Daniel's question about emerging options is really a question about which tradeoff to optimize for. And the answer is, it depends on what you're building. But the good news is that the options are getting better across the board. Streaming models are getting more accurate. Multilingual models are getting smaller. Tiny models are getting more capable.
The distillation work is the quiet revolution here. A large model trains a small model to mimic its behavior. The small model inherits most of the accuracy at a fraction of the size. That's how you get a forty million parameter model that transcribes twenty languages reasonably well. And it's why the parameter count is so misleading. A distilled forty million parameter model can outperform a non-distilled hundred million parameter model.
The parameter count was never the right metric. It was just the one that was easy to see. The real metrics are quantization, operator support, and thermal behavior. And those are the ones you have to dig for.
Which is why the Hugging Face workflow matters. The tags, the README, the quantization info, the operator list. Those are the signals that tell you whether a model will actually run on your phone. The parameter count is a distraction.
The runtime is the thing that ties it all together. Pick the wrong runtime and even a perfect model won't load. Pick the right runtime and a mediocre model will sing.
The practical advice is: start with AI Edge Gallery to see what your device can do. Then go to Hugging Face and filter by runtime tags. Read the READMEs. Check the quantization. Verify the operators. And don't trust desktop benchmarks.
The models that win on Android are the ones that ship with the best runtime support and the most forgiving operator sets. Not the ones with the highest accuracy on a desktop GPU.
Hilbert: My brother-in-law sells phones. He's been doing it since the two-thousands. He told me once the problem with dedicated silicon is the day they stop making it. You've got a drawer full of devices that worked fine until the chip revision changed. The firmware was locked to the specific revision. No update. No fix. Just a paperweight.
Hilbert: I worked a stint at a repair shop around then. We had a drawer full of dead smart devices. They all had dedicated DSPs for audio. The DSP firmware was locked to the chip revision. When the manufacturer moved to the next revision, the old firmware stopped getting updates. The devices still powered on. They just couldn't do the thing they were bought for.
Hilbert: When I hear TPU-optimized, I think, that's great until the TPU changes. The model you optimized for today's Tensor chip may not run on next year's. And you won't know until you try. The compatibility cliff is real.
Hilbert: I'm not saying don't use the TPU. I'm saying don't build your whole workflow around it. Keep a CPU path. Keep a portable model. Because the day they change the silicon, you'll be glad you did.
Hilbert: I've still got one of those dead DSP devices in a box in the garage. Been meaning to see if I can get it to boot. The box is labeled, do not open, two thousand seven.
A box labeled do not open from two thousand seven. That's not a box, that's a time capsule.
The compatibility cliff is a real concern. We're seeing it already with the vendor NPUs. Qualcomm's operator set is different from MediaTek's, which is different from Google's TPU. A model optimized for one doesn't carry to the others. And when the next generation of each arrives, the old optimizations may not carry forward either.
The question is whether the runtimes can abstract enough of this away. LiteRT and ExecuTorch are trying. The NPU delegate is supposed to handle the partitioning and the operator mapping. But the abstraction leaks. And every time it leaks, someone has to debug a model that worked yesterday and doesn't today.
The models that survive that churn are the ones with the most forgiving operator sets. The ones that don't depend on a specific silicon quirk. The ones that run acceptably on CPU and better on NPU. Those are the ones worth building on.
That's the open question. As Android gets more NPU variants, does the ecosystem fragment further, or do the runtimes abstract enough to keep models portable? I don't think we know yet. The next two or three years will decide it.
My bet is on fragmentation in the short term and abstraction in the long term. The runtimes will eventually catch up. But in the meantime, the practical advice is to keep a CPU fallback and test on real hardware.
The misconception people have is that parameter count is the deciding factor for what their Android device can run. It isn't. Quantization, operator compatibility, and runtime overhead all matter more. A thirty-nine million parameter model can be forty megabytes or a hundred and fifty, depending on how it's packaged.
A TPU-optimized model is not universally better. It's better on that specific TPU. On other hardware, it may not run at all. The optimization is a trade, not a free upgrade.
If you've tried running ASR on Android, leave a review and tell us what runtime and model you landed on. We're curious what the field looks like in practice.
Thanks to our producer, Hilbert Flumingtop, for keeping the show running.
This has been My Weird Prompts, the human-AI collaboration podcast.
We'll be back soon.
