AI Inference
549 episodes · Page 3 of 23
#6288: What Happens Between Model Checkpoints
Extending reinforcement learning on Olmo 3 Think for 21 days yielded a 20-point leap on IFBench without starting fresh pretraining.
#6287: Why Model Names Conceal Changes Behind the API
One API identifier spanned ten distinct releases over nineteen months without changing the name submitted by the customer.
#6278: Why Model Benchmarks Miss Small Workflow Friction
Switching Claude Code to Sonnet broke an everyday shell alias request that Opus solved simply by recognizing where to look in local context.
#6274: Machine Learning for Smells, Whale Codas, and Aftershocks
A graph neural network mapped molecular structures to scents, matching a consensus panel better than the typical individual human nose.
#6273: How Timestral Forecasts Time Series Data
Mistral's open-weight Timestral models predict up to 1,024 future steps zero-shot using grouped patches across twenty-one non-crossing quantiles.
#6272: Teaching Audio Models to Hear Beyond Words
Speech recognition rewards ignoring background noise, but audio-language models train specifically on environmental sounds and temporal scenes.
#6265: Why Upgrading Embedding Models Demands Reindexing
Cohere's Embed Five Fast queries against a Pro index reach 98.4 percent of performance without regenerating stored document vectors.
#6263: Why Numeric Data Lacks a Default Embedding Model
Popular time-series models like IBM's Granite TSPulse forecast trends rather than provide canonical vectors for similarity search.
#6257: Where Your Prompts Actually Go
Selecting US-only inference on Anthropic's direct API incurs a 1.1x multiplier across token categories, while global routing is the standard rate.
#6253: Why Context Windows Stalled at a Million Tokens
A dense 70B model requires about 328 gigabytes of KV cache to serve a single one-million-token prompt, exceeding the memory of an Nvidia B300 chip.
#6245: Why Inference Engines Change Benchmark Scores
A study found a 16.6 percentage point spread on GSM8K across different inference engines running identical weights on the same GPU.
#6243: Inside the Mixture of Experts Router
What actually decides which expert handles your token? A look inside the router, from sigmoid affinities to load balancing.
#6238: Which AI Models Actually Generate Good Training Data?
Open weights vs closed APIs for synthetic data, and why IFEval scores don't tell you which model will hold up.
#6237: Small Models, Big Questions: When Cheap Fables Get Expensive
Three million fables for pennies — but can a small model tell a good example from a plausible one?
#6228: What 12GB of VRAM Can Actually Do Now
Local AI on a mid-range AMD card: what fits, what's fast, and what's still out of reach in 2026.
#6223: How Many Epochs Does Fine-Tuning Actually Need?
Epochs are a receipt, not a plan. How to pick a training ceiling, read the curves, and know when to stop.
#6220: Mixture of Experts: Bigger Than It Looks
MoE models store a trillion parameters but only fire a few billion per token. Here's what the active number really measures — and why smaller isn't...
#6217: Fine-Tuning a Trillion-Parameter Model at Home
What does it actually take to full fine-tune a trillion-parameter model? Sixteen terabytes of optimizer state, hundreds of GPUs, and a very big bill.
#6216: What Do You Call the Middle Tier Model?
Sonnet, Sol, Flash, Medium — the daily-driver models nobody can name, and why they're squeezed from above and below.
#6211: Attention Is Still Winning: The Hybrid Model Census
Mamba layers shipped — inside hybrids. A census of Qwen, Nemotron, Kimi, Granite, and why MiniMax went back to full attention.
#6201: Inside Cloudflare's Clef: A Language Model That Can't Speak
Cloudflare's Clef reads text, images, and video but outputs only probabilities — no words. Here's what's actually under the hood.
#6198: Benchmarking a TTS Model on Your Own GPU
Real-time factor, warm-up artifacts, and why an 8-bit quantized speech model might not run any faster.
#6191: What Actually Happens During Fine-Tuning
Sixteen bytes per parameter in, two out — a walk through the forward pass, backward pass, and optimizer that turn a dataset into new weights.
#6190: When Your AI API Quietly Changes Under You
Same API name, different model behind it. How to tell when the thing you tested stops being the thing you're using.