AI Inference
549 episodes · Page 6 of 23
#5969: Ubuntu 26.04 LTS: Is Staying on LTS Still a Trade-Off?
Ubuntu 26.04 LTS shipped in April. Point releases, HWE kernels, and SRUs have quietly closed most of the freshness gap — but not everywhere.
#5964: Containers vs VMs for GPU Inference: What You Actually Pay
Containers cost 0.03%. Passthrough VMs cost 1%... or 36%. Same hardware, different systems — here's how to tell which you'll get.
#5963: Running One Model Across Mismatched GPUs
Tensor, pipeline, or offload? What actually happens when you run one model across several GPUs — including mismatched ones.
#5961: What Your GPU Is Actually Waiting On
Your GPU says 100% utilized. It's multiplying zeros. Here's what the counters actually measure during inference.
#5960: There's No Such Thing as Tokens Per Second
Why one tokens-per-second number can't describe an LLM server — and which metrics actually tell you if it's working.
#5959: What RTF Actually Tells You About Speech Models
Accuracy gets the headlines, but speed decides what ships. How RTF, latency percentiles, and memory shape real ASR deployments.
#5948: LangGraph, LangChain, and Deep Agents in 2026
Runtime, framework, harness — how three LangChain layers actually fit together, and what changed in 2026.
#5944: Keeping the Rollout Engine Fed: Inside RLHF Pipelines
Most of an RLHF run is spent waiting on generation, not gradients. Here's how TRL, OpenRLHF, and verl actually differ.
#5938: One Regex Rule Worked, One Was Impossible
A dictation app, two text-replacement rules, and the mathematical line between what regex can match and what it provably cannot.
#5936: Training a Model When You Don't Know the Labels Yet
Daniel wants to extract adapter specs from his shelf — but he can't name the labels until he's already annotating.
#5903: Video Tokens, Omni, and the Local VRAM Math
Video costs ~100 tokens per second. That single number explains why ChatGPT hides video upload and why local inference hits a KV cache wall.
#5900: Serving DeepSeek-V4.1-Flash on Your Own GPU
Five inference engines, one frontier MoE, and a shortlist that collapses depending on which GPU you already bought.
#5899: Same GPU, Same Weights, Different Answers
Identical hardware, identical weights, different engine — and answers that shift by up to 16.6 points.
#5898: ROCm 10.1: AMD's Software Stack Finally Grows Up
Three ROCm releases in six months — build system, agentic tooling, data movement. What actually changed for AMD GPU owners.
#5870: Homebox Fork's Two Backend Scripts Need a Real Job Runner
A Homebox fork has two incremental scripts — WebP conversion and AI image enrichment — and no scheduler built to run them safely.
#5860: Who Can Actually Train an LLM From Scratch?
Only a handful of organizations can truly train a model from random weights. Here's why nobody can count them.
#5859: Slow Labs, Fast Ships: What Release Cadence Actually Tells Us
Anthropic ships premium models every 22.5 days — the fastest lab in the industry. So much for the slow-and-careful story.
#5858: Routing AI Models: When 3 Rules Beat a Trained Router
A deep dive into AI model routing — and why a learned router may not beat three rules you wrote by hand.
#5857: Building a Scorecard for Our Own AI Writers
Daniel wants a real scorecard for the agent that writes this show — repetition, dialogue quality, and whether the retrieval harness actually works.
#5853: Who Actually Competes With Hugging Face?
2.96 million model repos, and 85.6% have under 200 downloads. What ModelScope, Kaggle, and Ollama actually do differently.
#5830: Serving Your Own Fine-Tuned Model in the Cloud
You fine-tuned an open-weight model. Now how does anyone actually talk to it? Dedicated GPUs vs serverless inference, and the math that decides it.
#5813: How Python Became AI's Default Language
Python was never designed for science. So how did a general-purpose scripting language end up carrying the entire AI industry?
#5804: Text Classifiers: Local Models vs Always-On Endpoints
A specialist classifier costs $5.70 per million messages. A general LLM costs $142. So why is the obvious product so hard to find?
#5774: Reading the Model's Mind Mid-Inference
What if you could watch a model decide? Inside the tools that open up inference mid-computation — and the limits they hit.