Hallucinations
306 episodes · Page 2 of 13
#6207: Two WERs, One Model: Inside ASR Eval Harnesses
Same model, same audio, two different word error rates. The gap lives in the normaliser — and that's just layer two of three.
#6190: When Your AI API Quietly Changes Under You
Same API name, different model behind it. How to tell when the thing you tested stops being the thing you're using.
#6189: What "Real" Improvement Actually Means in AI Evals
A p-value isn't the probability your result was luck. Here's what chance actually means in AI benchmarks — and why bold numbers need cross-examinat...
#6176: Do Audio Models Hear How You Said It?
A benchmark called LISTEN found audio models score near chance when emotion lives in the voice, not the words. What does that mean for real calls?
#6173: Auditing a Sycophantic Model From the Inside
A stumped lab, a fawning model, and the diagnostic playbook nobody has a name for — from eval slicing to persona vectors.
#6171: AI Audio Restoration: What Actually Ships in 2026
Noise removal and bandwidth extension are shipping products today. Matching a clip to a target EQ is still mostly a research prototype.
#6154: Who Checks the Labels? Weak vs Strong Supervision
Human labels, rule-based labels, or a teacher model writing the exam — what actually makes supervision "strong," and who verifies it?
#6149: 4 Hard Fails in 84 Answers: Testing AI Bias on Israel-Palestine
A 19-prompt, 6-model eval tested whether AI models push back evenly on false claims about Israel and Palestine. One model failed four times.
#6145: Do LLMs Nod Along With Conspiracy Theories?
A 98-prompt eval asks one question: when a believer brings a conspiracy theory, does the model push back, or help them dig?
#6142: We Gave 4 AI Models the Same Toast Prompt
Four models, one buttered-toast physics question. Three invented studies that never existed — and the oldest model was the most fun.
#6135: How Many Runs Does Your AI Eval Actually Need?
Reruns shrink one kind of noise and leave another untouched — so where should your eval budget really go?
#6096: How Long Does an M-Disc Actually Last?
A thousand years, or a hundred and eighty? What a burned disc on a Jerusalem patio can — and can't — tell us about archival storage.
#6056: Where Does Speaker Structure Belong in AI Podcast Scripts?
Label-and-parse pipelines break in ways that look like model errors but aren't. Plus what the research actually says about the format tax.
#6043: Write the Spec Before You Pick the Model
Long context, character voice, cheap at scale — three requirements that fight each other. The document that decides the fight.
#6042: Who Grades the Grader? AI Writing Benchmarks Put to the Test
Maths benchmarks can't see whether a model's prose collapses by chapter three. Five specialist leaderboards try — and our own writer is on them.
#6008: Why Codex Writes Code for a Dead API
Models don't hallucinate your API — they quote real docs from the wrong year. Here's why, and what actually fixes it.
#5970: Swapping Your Linux Kernel: What Actually Changes
Your kernel is swappable without reinstalling anything. But do Liquorix and XanMod actually beat the default? The benchmarks say less than the mark...
#5949: Inside Deep Research Agents: Claims, Citations, and the 84.7% Orchestrator Problem
Why 84.7% of citation errors originate at the orchestrator — and how a claim ledger fixes it.
#5946: Inside Humanity's Last Exam: 2,500 Questions AI Can't Ace
2,500 questions, ~1,000 experts, and a top score that jumped from 2.7% to 54.8% in 18 months. What is HLE actually measuring?
#5941: Five Minutes Reading Your Dictation Prompt
A listener's speech-to-text cleanup prompt, read out in full — then stress-tested against what ASR actually gets wrong.
#5936: Training a Model When You Don't Know the Labels Yet
Daniel wants to extract adapter specs from his shelf — but he can't name the labels until he's already annotating.
#5878: How a Sloth Learned to Talk: Corn's Origin Story
A neural interface gave Corn the bandwidth for speech. Nobody can explain where the accent, the manner, or Mongolia came from.
#5857: Building a Scorecard for Our Own AI Writers
Daniel wants a real scorecard for the agent that writes this show — repetition, dialogue quality, and whether the retrieval harness actually works.
#5851: The Earbud Assistant Problem Nobody Solves
The AI model is solved. So why does the assistant in your ear still fail? The answer is the unglamorous plumbing nobody demos.