vLLM
High-throughput LLM inference engine
Episodes
-
#6024: What's Actually Inside the KV Cache?890 bytes per token: why the KV cache is both what makes LLMs practical and what makes long context expensive. -
#6021: Why Chat Templates Break Models SilentlyEvery chat API flattens your messages into special tokens. Get one delimiter wrong and the model just gets quietly dumber. -
#6013: What Happens Inside One LLM Forward PassOne trip through the network, layer by layer — and why a classifier needs one pass while an LLM needs hundreds. -
#6011: Inside LLM Inference: Prefill, Decode, and the KV CacheOne 8K-token sequence on a 13B model eats 6.7GB of KV cache—while the weights are only 26GB. Here's what actually happens per request. -
#5991: Batch Inference on Your Own GPU: vLLM, SGLang, Ollama, llama.cppHosted batch APIs sell you half-price tokens. Here's how to get the same file-in, file-out workflow on your own Linux box. -
#5981: Batch Inference: Four Doors, Half PriceSame tokens, half the price — and almost nobody using it properly. A tour of batch inference from OpenAI to OpenRouter. -
#5964: Containers vs VMs for GPU Inference: What You Actually PayContainers cost 0.03%. Passthrough VMs cost 1%... or 36%. Same hardware, different systems — here's how to tell which you'll get. -
#5963: Running One Model Across Mismatched GPUsTensor, pipeline, or offload? What actually happens when you run one model across several GPUs — including mismatched ones. -
#5960: There's No Such Thing as Tokens Per SecondWhy one tokens-per-second number can't describe an LLM server — and which metrics actually tell you if it's working. -
#5944: Keeping the Rollout Engine Fed: Inside RLHF PipelinesMost of an RLHF run is spent waiting on generation, not gradients. Here's how TRL, OpenRLHF, and verl actually differ. -
#5900: Serving DeepSeek-V4.1-Flash on Your Own GPUFive inference engines, one frontier MoE, and a shortlist that collapses depending on which GPU you already bought. -
#5899: Same GPU, Same Weights, Different AnswersIdentical hardware, identical weights, different engine — and answers that shift by up to 16.6 points. -
#5898: ROCm 10.1: AMD's Software Stack Finally Grows UpThree ROCm releases in six months — build system, agentic tooling, data movement. What actually changed for AMD GPU owners. -
#5842: When Your TTS Model Updates ItselfResemble pushed new Chatterbox weights to Hugging Face. Our pipeline started using them without a single config change. -
#5830: Serving Your Own Fine-Tuned Model in the CloudYou fine-tuned an open-weight model. Now how does anyone actually talk to it? Dedicated GPUs vs serverless inference, and the math that decides it. -
#5481: Small Models, Big Guardrails: PII Detection in ChatGPTA tiny 600M parameter model is quietly scanning your ChatGPT tool calls for PII. Here's how that class of guardian model actually works. -
#5467: Small Models, Big Schemas: When JSON Constraints BackfireSmall models plus strict JSON schemas should be a safe bet. A 15,000-generation study found the opposite. -
#5436: Small Models as Rewriters, Not WritersWhy "don't say X" prompts backfire, and how a tiny grammar-constrained model can scrub a script without breaking its grammar. -
#5410: Adapters: 102KB That Reshapes a 403GB ModelA 102KB adapter file changes how a 403GB base model behaves — without ever merging into it. Here's how model adapters actually work. -
#5393: Fine-Tuning a Model on 100 Hand-Edited AnswersYou don't need 10,000 examples to make a model sound like you. The real number is closer to 100 — if the edits are opinionated. -
#5165: Weighting Memory in a RAG PipelineDaniel's memory leg won't fire. Turns out system prompts have no inherent vector-space privilege — but retrieval does have a real knob. -
#4973: Building a Pure AI Inference ServerWhy your "AI server" should look more like a database than a web app, and how to actually build one. -
#4970: Labels vs. Book Covers: Structured Extraction TradeoffsGeneral vision models vs. purpose-built pipelines for extracting data from messy real-world labels and covers. -
#4789: How LLMs Actually Know When to StopThree layers of stop signals, from EOS tokens to context limits — and why none of them really "know" when to stop. -
#4778: SSH MCP and Tailscale: Docker Migration GuideHow to configure Tailscale for Docker containers when migrating from public IPs to tailnet addresses. -
#4139: The Hosepipe-to-Digest AI: Your Outbox MiddlewareAn AI middleware that intercepts your outgoing emails, buffers them, and delivers a crisp executive summary. -
#3218: Building Your Own Cloud in 2026The software and hardware for a DIY private cloud have never been more feasible. Here's how to pick the right pieces. -
#2693: When AI Ignores Your Style GuideWhy your AI ignores formatting instructions and how to fix it with pipeline architecture, not model swaps. -
#2184: The Economics of Running AI AgentsProduction AI agents can cost $500K/month before optimization. Learn model routing, prompt caching, and token budgeting to cut costs 40-85% without... -
#2065: Why Run One AI When You Can Run Two?Speculative decoding makes LLMs 2-3x faster with zero quality loss by using a small draft model to guess tokens that a large model verifies in para... -
#2061: The Memory Bottleneck That Drives Attention DesignAttention is the engine of modern AI, but it’s also a memory hog. Here’s how MQA, GQA, and MLA evolved to fix it. -
#2040: The Rebellion Against Big Tech's AI Lock-InWhy run LLMs locally? We break down Ollama, llama.cpp, vLLM, and llamafile—and when to use each. -
#2005: Beyond Vibes: The Hard Science of LLM EvaluationRunning the same LLM on different GPUs can produce different results. Here’s why that happens and how to test for it. -
#1544: Why It Costs More to Run AI Than to Build ItDiscover why the AI runtime is the unsung hero of the tech stack, determining whether your AI feels like a snappy conversation or a slow crawl. -
#1081: The K-V Cache: Solving AI’s Invisible Memory TaxWhy does your AI get slower as you chat? Discover the K-V cache, the invisible bottleneck of generative AI, and how we're fixing it in 2026.