Inference and Serving
How models are served and paid for: batch APIs, inference providers, serverless GPUs, latency and the KV cache.
- 1
Renting vs. Building: The Hidden Choices in AI Inference
Ever wonder how AI magic happens? We demystify AI inference, exploring where and how models truly operate.
- 2
Decoding the Transformer: From Attention to Inference
Herman and Corn dive into the mechanics of transformer inference, exploring how models turn massive matrices into meaningful conversation.
- 3
The Silicon Sharing Economy: Inside Serverless GPUs
How do small teams run massive AI models without $50,000 chips? Corn and Herman dive into the hidden plumbing of serverless GPU providers.
- 4
The Speed of Thought: Inside the New Era of Inference
The war for model size is over. Explore the engineering breakthroughs making massive AI models faster than human thought.
- 5
The War Against Latency: Engineering Real-Time AI
From KV cache monsters to sub-100ms response times, explore the hardware and software innovations making real-time AI a reality.
- 6
Claude's Latency Profile and SLA Guarantees
Claude is measurably slower than competitors—and Anthropic's SLA promises are even thinner than the latency numbers suggest. What enterprises actua...
- 7
Batch APIs: The 50% Discount You're Probably Misusing
Batch inference APIs offer 50% off — but only for the right workloads. Here's when they actually make sense.
- 8
Measuring AI API Latency Through the Black Box
How to benchmark token throughput and debug slowdowns in closed CLI tools like Claude Code using OpenTelemetry and mitmproxy.
- 9
Why Instructional Models Beat Conversational for Batch AI
Beyond cheaper tokens—how batch inference changes AI workflows and why instructional models beat conversational ones for automated jobs.
- 10
GPU Idle Waste and Serverless Green Computing
Why your dedicated GPU burns 130 watts doing nothing, and how serverless platforms cut energy waste by more than half.
- 11
The Hidden Stateful Side of Serverless GPU
How Modal, RunPod, and other platforms handle container builds, caching, and versioning under the hood.
- 12
CPU vs GPU Inference: When to Skip the GPU
Why running AI on CPU isn't a toy — and when the GPU advantage vanishes.
- 13
Building a Pure AI Inference Server
Why your "AI server" should look more like a database than a web app, and how to actually build one.
- 14
Two Queues, One Cron Tick: Serverless Batch Pipelines
What does a queue look like when your process is disposable? Two queues, one cron tick, and a 50% batch discount.
- 15
Serving Your Own Fine-Tuned Model in the Cloud
You fine-tuned an open-weight model. Now how does anyone actually talk to it? Dedicated GPUs vs serverless inference, and the math that decides it.
- 16
Containers vs VMs for GPU Inference: What You Actually Pay
Containers cost 0.03%. Passthrough VMs cost 1%... or 36%. Same hardware, different systems — here's how to tell which you'll get.
- 17
There's No Such Thing as Tokens Per Second
Why one tokens-per-second number can't describe an LLM server — and which metrics actually tell you if it's working.
- 18
Running One Model Across Mismatched GPUs
Tensor, pipeline, or offload? What actually happens when you run one model across several GPUs — including mismatched ones.
- 19
What a $1M AI Inference Contract Actually Contains
The sticker price isn't the deal. Inside the commit floors, discount curves, and overage traps of enterprise AI contracts.
- 20
Batch Inference: Four Doors, Half Price
Same tokens, half the price — and almost nobody using it properly. A tour of batch inference from OpenAI to OpenRouter.
- 21
What Enterprises Actually Pay for AI Inference
Twelve and a half billion dollars in enterprise API spend — and nobody can tell you who spent it. Inside AI's most undocumented market.
- 22
What Do Inference Providers Actually Sell?
Same open weights, 20x cost gap. What Fireworks, Together, and Groq actually sell — and when to just rent a GPU.
- 23
Data Residency vs. Inference Residency: What the Law Actually Says
GDPR doesn't require local GPUs. Here's what actually drives geographically restricted AI inference — and the three layers everyone conflates.
- 24
Inside LLM Inference: Prefill, Decode, and the KV Cache
One 8K-token sequence on a 13B model eats 6.7GB of KV cache—while the weights are only 26GB. Here's what actually happens per request.
- 25
Batch Inference on Your Own GPU: vLLM, SGLang, Ollama, llama.cpp
Hosted batch APIs sell you half-price tokens. Here's how to get the same file-in, file-out workflow on your own Linux box.
- 26
Nobody Can Buy Unlimited AI Inference
Every published tier, every reserved deal, every prepaid reservation — all bounded. Even Azure says it out loud: quota isn't capacity.