SGLang
Serving framework with constrained decoding
Episodes
-
#6201: Inside Cloudflare's Clef: A Language Model That Can't SpeakCloudflare's Clef reads text, images, and video but outputs only probabilities — no words. Here's what's actually under the hood. -
#5991: Batch Inference on Your Own GPU: vLLM, SGLang, Ollama, llama.cppHosted batch APIs sell you half-price tokens. Here's how to get the same file-in, file-out workflow on your own Linux box. -
#5981: Batch Inference: Four Doors, Half PriceSame tokens, half the price — and almost nobody using it properly. A tour of batch inference from OpenAI to OpenRouter. -
#5900: Serving DeepSeek-V4.1-Flash on Your Own GPUFive inference engines, one frontier MoE, and a shortlist that collapses depending on which GPU you already bought. -
#5899: Same GPU, Same Weights, Different AnswersIdentical hardware, identical weights, different engine — and answers that shift by up to 16.6 points. -
#5898: ROCm 10.1: AMD's Software Stack Finally Grows UpThree ROCm releases in six months — build system, agentic tooling, data movement. What actually changed for AMD GPU owners. -
#5830: Serving Your Own Fine-Tuned Model in the CloudYou fine-tuned an open-weight model. Now how does anyone actually talk to it? Dedicated GPUs vs serverless inference, and the math that decides it. -
#5436: Small Models as Rewriters, Not WritersWhy "don't say X" prompts backfire, and how a tiny grammar-constrained model can scrub a script without breaking its grammar. -
#4973: Building a Pure AI Inference ServerWhy your "AI server" should look more like a database than a web app, and how to actually build one.