TensorRT-LLM
NVIDIA's fastest inference engine, complex to build
Episodes
-
#5963: Running One Model Across Mismatched GPUsTensor, pipeline, or offload? What actually happens when you run one model across several GPUs — including mismatched ones. -
#5960: There's No Such Thing as Tokens Per SecondWhy one tokens-per-second number can't describe an LLM server — and which metrics actually tell you if it's working. -
#5900: Serving DeepSeek-V4.1-Flash on Your Own GPUFive inference engines, one frontier MoE, and a shortlist that collapses depending on which GPU you already bought. -
#5899: Same GPU, Same Weights, Different AnswersIdentical hardware, identical weights, different engine — and answers that shift by up to 16.6 points. -
#4973: Building a Pure AI Inference ServerWhy your "AI server" should look more like a database than a web app, and how to actually build one. -
#2456: Choosing Between AI Cloud ProvidersA practical guide to choosing between Modal, RunPod, Nebius, and Baseten for AI workloads. -
#2065: Why Run One AI When You Can Run Two?Speculative decoding makes LLMs 2-3x faster with zero quality loss by using a small draft model to guess tokens that a large model verifies in para... -
#2061: The Memory Bottleneck That Drives Attention DesignAttention is the engine of modern AI, but it’s also a memory hog. Here’s how MQA, GQA, and MLA evolved to fix it. -
#2041: The "MPEG Moment" for AI: Llamafile & Native ModelsWhy are we squeezing massive cloud models onto desktops? Meet the "native" AI revolution. -
#2005: Beyond Vibes: The Hard Science of LLM EvaluationRunning the same LLM on different GPUs can produce different results. Here’s why that happens and how to test for it. -
#1544: Why It Costs More to Run AI Than to Build ItDiscover why the AI runtime is the unsung hero of the tech stack, determining whether your AI feels like a snappy conversation or a slow crawl.