AI Evaluation
513 episodes · Page 4 of 22
#5959: What RTF Actually Tells You About Speech Models
Accuracy gets the headlines, but speed decides what ships. How RTF, latency percentiles, and memory shape real ASR deployments.
#5957: The Scientific Plugins Hiding in Plain Sight
ChatGPT's plugin directory has 5,723 entries — but the real scientific AI work is happening in arXiv papers, not plugins.
#5949: Inside Deep Research Agents: Claims, Citations, and the 84.7% Orchestrator Problem
Why 84.7% of citation errors originate at the orchestrator — and how a claim ledger fixes it.
#5946: Inside Humanity's Last Exam: 2,500 Questions AI Can't Ace
2,500 questions, ~1,000 experts, and a top score that jumped from 2.7% to 54.8% in 18 months. What is HLE actually measuring?
#5945: How a Reward Model Actually Trains an LLM
One number, no gradients, four models in the room. What a reward model actually does inside the RLHF loop.
#5941: Five Minutes Reading Your Dictation Prompt
A listener's speech-to-text cleanup prompt, read out in full — then stress-tested against what ASR actually gets wrong.
#5937: The Fine-Tuning Career: Titles, Skills, and What It Pays
The job above the labeling layer exists — it just doesn't have one name. What it takes, what it pays, and how to sell it.
#5922: Teaching a Model Your Voice: SFT vs DPO
Three ways to teach a model your writing voice — and the evidence says one of them quietly makes things worse.
#5917: The Women Who Were Computers Before Computers Existed
Before machines did the math, people did — and most of them were women. Reconstructing the desk of the human computer.
#5915: How Multi-Agent AI Simulations Keep a Shared World State
Concordia keeps the world in a dictionary an LLM writes to. Generative Agents keeps it in a spreadsheet. CAMEL keeps it in the chat log.
#5903: Video Tokens, Omni, and the Local VRAM Math
Video costs ~100 tokens per second. That single number explains why ChatGPT hides video upload and why local inference hits a KV cache wall.
#5899: Same GPU, Same Weights, Different Answers
Identical hardware, identical weights, different engine — and answers that shift by up to 16.6 points.
#5859: Slow Labs, Fast Ships: What Release Cadence Actually Tells Us
Anthropic ships premium models every 22.5 days — the fastest lab in the industry. So much for the slow-and-careful story.
#5858: Routing AI Models: When 3 Rules Beat a Trained Router
A deep dive into AI model routing — and why a learned router may not beat three rules you wrote by hand.
#5857: Building a Scorecard for Our Own AI Writers
Daniel wants a real scorecard for the agent that writes this show — repetition, dialogue quality, and whether the retrieval harness actually works.
#5845: The $700 Pin and Other AI Flops
Five AI misfires, from a $700 pin that overheats doing nothing to a flagship model nobody remembers.
#5842: When Your TTS Model Updates Itself
Resemble pushed new Chatterbox weights to Hugging Face. Our pipeline started using them without a single config change.
#5832: When Your GPS Is Lying: Aviation's Spoofing Problem
GPS spoofing is feeding pilots fake positions — and the panel stays green. How crews detect it, and why the airport stand sign can't save you.
#5817: When Agents Don't Need Your Pretty Buttons
Agents are becoming major software users — and they prefer raw command-line tools to polished interfaces. What replaces UX when your user can't see?
#5805: Freeze the Model, Move the Labels
Your classifier isn't stale — your label set moved. Why retraining is cheap, and why not retraining might be smarter.
#5804: Text Classifiers: Local Models vs Always-On Endpoints
A specialist classifier costs $5.70 per million messages. A general LLM costs $142. So why is the obvious product so hard to find?
#5799: Two Agents, One Approval Gate: Wiring a Dictation Workflow
A dictated prompt becomes an episode — if one agent resolves facts, a second tightens prose, and a human approves the dispatch.
#5777: Lorebooks: The Hobbyist RAG That Beat the Labs
How a chat frontend fork built a full retrieval system — keywords, vectors, recursion — so fictional characters would stop forgetting who they are.
#5770: Talking to Your Data: What MCP Doesn't Solve
MCP standardized the pipe. It has no opinion about what flows through it — and that's where the hard part lives.