AI Inference
549 episodes · Page 8 of 23
#5408: When Small NLP Models Beat the LLM
Feature extraction, fill-mask, token classification — the classic NLP tasks still have a job. Here's when a small model beats a frontier API.
#5397: Chaining Small Models for Voice Cleanup
Six cleanup stages at 97% accuracy each compound to 83% end-to-end. So how many small models can you actually chain?
#5391: The Software Behind Urgent Care Triage
Big buttons, emoji vitals, and a system that says "order IV" — what's actually running behind the triage screen?
#5390: Chaining Small Models for Dictation Cleanup
Daniel's Android dictation fork won't render "three point five" as a decimal. How many models does cleanup actually need?
#5384: Android ASR Runtimes: LiteRT, ExecuTorch, and Why Your Phone Has No VRAM
Why does your phone have no VRAM number? A tour of Android's runtime layer and what it takes to run ASR locally.
#5345: Forecasting for Cranes, Fires, and 15-Minute Decisions
The public five-day forecast is a loss leader. Here's what bespoke weather looks like when money and lives ride on it.
#5327: Catching Duplicate Submissions Before They Burn Credits
A dropped webhook response caused a double submission. Here's how to build a screener that catches duplicates before generation runs.
#5301: Why Talking Robots Are Really a Committee
That humanoid chatting while it moves? It's not one brain — it's a stack of separate models glued together.
#5285: Two Queues, One Cron Tick: Serverless Batch Pipelines
What does a queue look like when your process is disposable? Two queues, one cron tick, and a 50% batch discount.
#5230: Why Your Research Agent Won't Stop Searching
Most agent search runs add no new evidence — and 66-85% of the context window is just search snippets. Here's what actually stops the loop.
#5188: DeepSeek's Point Release That Isn't
DeepSeek shipped a whole new architecture and called it a point release. Here's what actually changed inside the model.
#5184: AI Is a Number Factory, Not a Wordsmith
Under the prose, every AI model is just matrices of floating-point math. So where does the randomness actually come from?
#5182: DeepSeek V4.1 Flash: 1M Context, 437x Smaller KV Cache
DeepSeek V4.1 Flash landed with a 1M-token window and a KV cache 437x smaller than V1. Here's what actually changed — and why the middle of your co...
#5154: DeepSeek's Two-Endpoint Philosophy
DeepSeek quietly routed Pro traffic to Flash — and that routing change says everything about its two-endpoint strategy.
#5044: Abilene's AI Arms Race: Power, Water, and Backlash
Why a Texas town of 100,000 is hosting the biggest AI build ever—and why locals are starting to push back.
#5041: GPT-6 Astra, 100K GPUs, and the AGI Question
OpenAI's GPT-6 Astra arrives with an AGI claim and a 100K-GPU training run. We break down what Stargate LLC actually is and whether the scale justi...
#5004: What Precision Is Your Model Actually Running At?
When you call an API, are you getting the model you think you are? Inference providers quantize more than they admit.
#4973: Building a Pure AI Inference Server
Why your "AI server" should look more like a database than a web app, and how to actually build one.
#4972: CPU vs GPU Inference: When to Skip the GPU
Why running AI on CPU isn't a toy — and when the GPU advantage vanishes.
#4970: Labels vs. Book Covers: Structured Extraction Tradeoffs
General vision models vs. purpose-built pipelines for extracting data from messy real-world labels and covers.
#4967: AI Gateways: Worth the Middleware Tax?
When does an AI gateway's overhead justify its features? We break down the tradeoffs.
#4966: Local vs Cloud: Running Hugging Face Models
Hugging Face's compatibility tracker, cache management, and the real difference between Inference Endpoints and the direct API.
#4958: Multi-Model Transcription with LLM Reconciliation
Using multiple transcription engines and an LLM judge to catch errors in voice-to-text — especially tricky technical terms.
#4920: Local Dictation on Android: The Real Bottlenecks
Why on-device speech-to-text on Android hits a wall at 30 seconds — and what silicon actually matters.