vLLM
open-source software for large language model inference
Episodes
-
#5830: Serving Your Own Fine-Tuned Model in the CloudYou fine-tuned an open-weight model. Now how does anyone actually talk to it? Dedicated GPUs vs serverless inference, and the math that decides it. -
#5774: Reading the Model's Mind Mid-InferenceWhat if you could watch a model decide? Inside the tools that open up inference mid-computation — and the limits they hit. -
#5481: Small Models, Big Guardrails: PII Detection in ChatGPTA tiny 600M parameter model is quietly scanning your ChatGPT tool calls for PII. Here's how that class of guardian model actually works. -
#5467: Small Models, Big Schemas: When JSON Constraints BackfireSmall models plus strict JSON schemas should be a safe bet. A 15,000-generation study found the opposite. -
#5436: Small Models as Rewriters, Not WritersWhy "don't say X" prompts backfire, and how a tiny grammar-constrained model can scrub a script without breaking its grammar. -
#5410: Adapters: 102KB That Reshapes a 403GB ModelA 102KB adapter file changes how a 403GB base model behaves — without ever merging into it. Here's how model adapters actually work. -
#5393: Fine-Tuning a Model on 100 Hand-Edited AnswersYou don't need 10,000 examples to make a model sound like you. The real number is closer to 100 — if the edits are opinionated. -
#5165: Weighting Memory in a RAG PipelineDaniel's memory leg won't fire. Turns out system prompts have no inherent vector-space privilege — but retrieval does have a real knob.