AI Inference
549 episodes · Page 5 of 23
#6034: Claude Opus 5.5: What Anthropic Actually Shipped
Anthropic's new flagship is cheaper than the model it replaces — and won't let you turn thinking off.
#6029: One Model, Two Modes: Does AI Consolidation Hold Up?
OpenAI hides a router, DeepSeek merged two modes, Anthropic removed the knob. Is the model landscape really consolidating?
#6028: ASR vs Multimodal LLMs: What Actually Won
Dedicated ASR plus a cleanup model is quietly beating end-to-end dictation on the hardware you already own.
#6024: What's Actually Inside the KV Cache?
890 bytes per token: why the KV cache is both what makes LLMs practical and what makes long context expensive.
#6021: Why Chat Templates Break Models Silently
Every chat API flattens your messages into special tokens. Get one delimiter wrong and the model just gets quietly dumber.
#6016: What 70 Billion Parameters Actually Means
A single artificial neuron, built from scratch — then the hard part: does that neuron analogy still describe a real transformer?
#6014: Why Every AI Model Is Secretly a Decoder
Encoder, decoder, or both? The three transformer shapes, why one won, and why BERT still refuses to die.
#6013: What Happens Inside One LLM Forward Pass
One trip through the network, layer by layer — and why a classifier needs one pass while an LLM needs hundreds.
#6012: Inside the Decode Loop: How LLMs Generate One Token at a Time
The model doesn't choose tokens or decide when to stop — the engine does. A step-by-step tour of one decode iteration.
#6011: Inside LLM Inference: Prefill, Decode, and the KV Cache
One 8K-token sequence on a 13B model eats 6.7GB of KV cache—while the weights are only 26GB. Here's what actually happens per request.
#6009: Vertex AI Is Now the Gemini Agent Platform
Vertex AI just became the Gemini Enterprise Agent Platform. Here's the family tree, the five products, and when to actually move.
#6007: What a $1M AI Inference Contract Actually Contains
The sticker price isn't the deal. Inside the commit floors, discount curves, and overage traps of enterprise AI contracts.
#6006: Data Residency vs. Inference Residency: What the Law Actually Says
GDPR doesn't require local GPUs. Here's what actually drives geographically restricted AI inference — and the three layers everyone conflates.
#6005: What's Actually Behind a Government AI Chatbot
Singapore runs open-weight models on commercial cloud. The UK built its own chatbot, then shut it down for Copilot. What's really behind civil-serv...
#6004: Nobody Can Buy Unlimited AI Inference
Every published tier, every reserved deal, every prepaid reservation — all bounded. Even Azure says it out loud: quota isn't capacity.
#6003: What Enterprises Actually Pay for AI Inference
Twelve and a half billion dollars in enterprise API spend — and nobody can tell you who spent it. Inside AI's most undocumented market.
#6002: Claude on AWS: Three Doors, One Rate Card
Same per-token price everywhere — so why buy Claude through AWS instead of Anthropic? The answer is commitments, compliance, and silicon.
#6001: What Do Inference Providers Actually Sell?
Same open weights, 20x cost gap. What Fireworks, Together, and Groq actually sell — and when to just rent a GPU.
#6000: 6,000 Episodes and a GPU Bill: Inside My Weird Prompts
Six thousand episodes, one human in the loop, and a Modal bill that keeps growing. A behind-the-scenes update on the podcast.
#5995: NVIDIA ASR: Parakeet, Canary, and Nemotron Explained
Whisper gives you one dial. NVIDIA gives you five. A practical map of the Parakeet, Canary, and Nemotron speech models.
#5991: Batch Inference on Your Own GPU: vLLM, SGLang, Ollama, llama.cpp
Hosted batch APIs sell you half-price tokens. Here's how to get the same file-in, file-out workflow on your own Linux box.
#5981: Batch Inference: Four Doors, Half Price
Same tokens, half the price — and almost nobody using it properly. A tour of batch inference from OpenAI to OpenRouter.
#5975: Merging vs Distillation: Can Two Models Become One?
Merging and distillation get lumped together, but they're opposites. One composes existing capability; the other has to teach it.
#5974: One Model or Two? ASR Cleanup Explained
If speech models can punctuate and capitalize on their own, why does a separate 484MB cleanup model exist at all?