Series

Inference and Serving

How models are served and paid for: batch APIs, inference providers, serverless GPUs, latency and the KV cache.

26 episodes
  1. 1

    Renting vs. Building: The Hidden Choices in AI Inference

    Ever wonder how AI magic happens? We demystify AI inference, exploring where and how models truly operate.

    26:20 · 10 Dec 2025
  2. 2

    Decoding the Transformer: From Attention to Inference

    Herman and Corn dive into the mechanics of transformer inference, exploring how models turn massive matrices into meaningful conversation.

    19:37 · 21 Jan 2026
  3. 3

    The Silicon Sharing Economy: Inside Serverless GPUs

    How do small teams run massive AI models without $50,000 chips? Corn and Herman dive into the hidden plumbing of serverless GPU providers.

    24:32 · 4 Feb 2026
  4. 4

    The Speed of Thought: Inside the New Era of Inference

    The war for model size is over. Explore the engineering breakthroughs making massive AI models faster than human thought.

    20:55 · 23 Mar 2026
  5. 5

    The War Against Latency: Engineering Real-Time AI

    From KV cache monsters to sub-100ms response times, explore the hardware and software innovations making real-time AI a reality.

    23:47 · 26 Mar 2026
  6. 6

    Claude's Latency Profile and SLA Guarantees

    Claude is measurably slower than competitors—and Anthropic's SLA promises are even thinner than the latency numbers suggest. What enterprises actua...

    24:57 · 12 Apr 2026
  7. 7

    Batch APIs: The 50% Discount You're Probably Misusing

    Batch inference APIs offer 50% off — but only for the right workloads. Here's when they actually make sense.

    25:15 · 26 Apr 2026
  8. 8

    Measuring AI API Latency Through the Black Box

    How to benchmark token throughput and debug slowdowns in closed CLI tools like Claude Code using OpenTelemetry and mitmproxy.

    26:16 · 29 Apr 2026
  9. 9

    Why Instructional Models Beat Conversational for Batch AI

    Beyond cheaper tokens—how batch inference changes AI workflows and why instructional models beat conversational ones for automated jobs.

    38:13 · 5 May 2026
  10. 10

    GPU Idle Waste and Serverless Green Computing

    Why your dedicated GPU burns 130 watts doing nothing, and how serverless platforms cut energy waste by more than half.

    30:36 · 12 May 2026
  11. 11

    The Hidden Stateful Side of Serverless GPU

    How Modal, RunPod, and other platforms handle container builds, caching, and versioning under the hood.

    32:27 · 12 May 2026
  12. 12

    CPU vs GPU Inference: When to Skip the GPU

    Why running AI on CPU isn't a toy — and when the GPU advantage vanishes.

    27:57 · 1 Sept 2026
  13. 13

    Building a Pure AI Inference Server

    Why your "AI server" should look more like a database than a web app, and how to actually build one.

    26:50 · 1 Sept 2026
  14. 14

    Two Queues, One Cron Tick: Serverless Batch Pipelines

    What does a queue look like when your process is disposable? Two queues, one cron tick, and a 50% batch discount.

    22:54 · 17 Sept 2026
  15. 15

    Serving Your Own Fine-Tuned Model in the Cloud

    You fine-tuned an open-weight model. Now how does anyone actually talk to it? Dedicated GPUs vs serverless inference, and the math that decides it.

    24:20 · 10 Oct 2026
  16. 16

    Containers vs VMs for GPU Inference: What You Actually Pay

    Containers cost 0.03%. Passthrough VMs cost 1%... or 36%. Same hardware, different systems — here's how to tell which you'll get.

    30:37 · 14 Oct 2026
  17. 17

    There's No Such Thing as Tokens Per Second

    Why one tokens-per-second number can't describe an LLM server — and which metrics actually tell you if it's working.

    30:16 · 14 Oct 2026
  18. 18

    Running One Model Across Mismatched GPUs

    Tensor, pipeline, or offload? What actually happens when you run one model across several GPUs — including mismatched ones.

    31:32 · 14 Oct 2026
  19. 19

    What a $1M AI Inference Contract Actually Contains

    The sticker price isn't the deal. Inside the commit floors, discount curves, and overage traps of enterprise AI contracts.

    26:36 · 15 Oct 2026
  20. 20

    Batch Inference: Four Doors, Half Price

    Same tokens, half the price — and almost nobody using it properly. A tour of batch inference from OpenAI to OpenRouter.

    34:14 · 15 Oct 2026
  21. 21

    What Enterprises Actually Pay for AI Inference

    Twelve and a half billion dollars in enterprise API spend — and nobody can tell you who spent it. Inside AI's most undocumented market.

    25:19 · 15 Oct 2026
  22. 22

    What Do Inference Providers Actually Sell?

    Same open weights, 20x cost gap. What Fireworks, Together, and Groq actually sell — and when to just rent a GPU.

    26:13 · 15 Oct 2026
  23. 23

    Data Residency vs. Inference Residency: What the Law Actually Says

    GDPR doesn't require local GPUs. Here's what actually drives geographically restricted AI inference — and the three layers everyone conflates.

    29:49 · 15 Oct 2026
  24. 24

    Inside LLM Inference: Prefill, Decode, and the KV Cache

    One 8K-token sequence on a 13B model eats 6.7GB of KV cache—while the weights are only 26GB. Here's what actually happens per request.

    34:39 · 15 Oct 2026
  25. 25

    Batch Inference on Your Own GPU: vLLM, SGLang, Ollama, llama.cpp

    Hosted batch APIs sell you half-price tokens. Here's how to get the same file-in, file-out workflow on your own Linux box.

    21:12 · 15 Oct 2026
  26. 26

    Nobody Can Buy Unlimited AI Inference

    Every published tier, every reserved deal, every prepaid reservation — all bounded. Even Azure says it out loud: quota isn't capacity.

    21:23 · 15 Oct 2026