#ai-inference
38 episodes
#5447: The Models That Never Talk Back
Some models read text, score it, and return a number. No chat, no reasoning, just decisions — and they're running under every router you use.
#5391: The Software Behind Urgent Care Triage
Big buttons, emoji vitals, and a system that says "order IV" — what's actually running behind the triage screen?
#5285: Two Queues, One Cron Tick: Serverless Batch Pipelines
What does a queue look like when your process is disposable? Two queues, one cron tick, and a 50% batch discount.
#5188: DeepSeek's Point Release That Isn't
DeepSeek shipped a whole new architecture and called it a point release. Here's what actually changed inside the model.
#5184: AI Is a Number Factory, Not a Wordsmith
Under the prose, every AI model is just matrices of floating-point math. So where does the randomness actually come from?
#5182: DeepSeek V4.1 Flash: 1M Context, 437x Smaller KV Cache
DeepSeek V4.1 Flash landed with a 1M-token window and a KV cache 437x smaller than V1. Here's what actually changed — and why the middle of your co...
#5167: The Stethoscope Isn't Dead, It's Getting AI Ears
Two centuries of the stethoscope — and why AI is now making it more useful than ever, not obsolete.
#5004: What Precision Is Your Model Actually Running At?
When you call an API, are you getting the model you think you are? Inference providers quantize more than they admit.
#4973: Building a Pure AI Inference Server
Why your "AI server" should look more like a database than a web app, and how to actually build one.
#2467: The Time Tax on API Access
How OpenAI and Anthropic structure API tiers, rate limits, and why your billing history matters more than you think.
#2464: Batch APIs: The 50% Discount You're Probably Misusing
Batch inference APIs offer 50% off — but only for the right workloads. Here's when they actually make sense.
#2456: Choosing Between AI Cloud Providers
A practical guide to choosing between Modal, RunPod, Nebius, and Baseten for AI workloads.
#2431: The 3 Markets in an AI Trench Coat
GPUs, LPUs, and ASICs: why the best hardware for AI depends entirely on what you're trying to do.
#2254: How to Test an AI Pipeline Change
When you tweak one part of a complex AI agent system, how do you know if it actually improved anything? The answer lies in engineering checkpoints.
#2249: Building Custom Benchmarks for Agentic Systems
Public benchmarks fail for agentic systems. Learn how to build evaluation frameworks that actually predict production behavior.
#2243: What Enterprise AI Pricing Actually Negotiates
Enterprise customers rarely get the deep discounts they expect from AI APIs. What they actually negotiate for—and why the ramp-up requirement exist...
#2214: The Three Failure Modes of AI News Systems
When a conflict changes hourly, AI systems built for yesterday's information fail. Here's how to architect pipelines that actually keep up.
#2184: The Economics of Running AI Agents
Production AI agents can cost $500K/month before optimization. Learn model routing, prompt caching, and token budgeting to cut costs 40-85% without...
#2179: Building Cost-Resilient AI Agents
Failed API calls in agent loops aren't just technical problems—they're direct budget drains. Here's how checkpointing, retry strategies, and cachin...
#2160: Claude's Latency Profile and SLA Guarantees
Claude is measurably slower than competitors—and Anthropic's SLA promises are even thinner than the latency numbers suggest. What enterprises actua...
#2123: Human Reaction Time vs. AI Latency
We obsess over shaving milliseconds off AI response times, but human biology has a hard limit. Here’s why your brain can’t keep up.
#2115: Why AI Answers Differ Even When You Ask Twice
You ask an AI the same question twice and get two different answers. It’s not a bug—it’s physics.
#2065: Why Run One AI When You Can Run Two?
Speculative decoding makes LLMs 2-3x faster with zero quality loss by using a small draft model to guess tokens that a large model verifies in para...
#2060: The Tokenizer's Hidden Tax on Non-English Text
Why does a simple greeting in Mandarin cost more to process than in English? It's the tokenizer's hidden inefficiency.