AI Evaluation
512 episodes
#6194: Teaching a TTS Model His Own Accent
One man fine-tuned a text-to-speech model on his own voice to get Irish-accented English that can say Hebrew words mid-sentence.
#6192: What Actually Happens When You Train a Model
Data in, model out — except the "data" is pairs, the split is the whole game, and one leaked speaker can fake your score.
#6190: When Your AI API Quietly Changes Under You
Same API name, different model behind it. How to tell when the thing you tested stops being the thing you're using.
#6189: What "Real" Improvement Actually Means in AI Evals
A p-value isn't the probability your result was luck. Here's what chance actually means in AI benchmarks — and why bold numbers need cross-examinat...
#6185: Why AI Labs Sell a Tier Above the Flagship
Fable, Deep Think, SuperGrok Heavy — what the $300/month top tier actually does that the flagship can't.
#6180: Mapping Code-Switching: 113 Pairs, No Community
Researchers count 113 code-switched language pairs — but a pair is just a label on a file, not a community of speakers.
#6176: Do Audio Models Hear How You Said It?
A benchmark called LISTEN found audio models score near chance when emotion lives in the voice, not the words. What does that mean for real calls?
#6175: Video Models Can't Tell Time: What Motion Benchmarks Reveal
Frontier models score 48% on tasks where humans hit 98% — the gap is motion, speed, and sequence, not scene description.
#6174: What Your Voice Reveals: Accent, Distance, and Movement in Audio AI
What can AI actually hear in your voice? Accent detection, speaker distance, whale songs, and the gap between deployed audio models and chatty gene...
#6173: Auditing a Sycophantic Model From the Inside
A stumped lab, a fawning model, and the diagnostic playbook nobody has a name for — from eval slicing to persona vectors.
#6172: Recipes vs. Regeneration: AI Photo Editing in 2026
Two ways AI edits photos now: models that write a readable recipe of slider values, and models that just repaint the whole frame.
#6171: AI Audio Restoration: What Actually Ships in 2026
Noise removal and bandwidth extension are shipping products today. Matching a clip to a target EQ is still mostly a research prototype.
#6169: What It's Like to Be a Token Inside an LLM
A token's-eye view of an LLM's forward pass — no cone, no squeezing, just a hallway where every layer scribbles on you.
#6164: We Gave Each Host Its Own AI Agent
We split our two hosts into separate AI agents with private context windows. The bill came to 78 cents an episode.
#6159: When a Speech Model Can't Decide How to Write Dollars
One dataset says "$20," another says "twenty dollars." Fine-tuning a speech model means finding out which convention wins — and why it matters.
#6158: Naming the ASR Sub-Tasks Nobody Mapped
A 2001 linguistics paper already named Daniel's problem — but it points the other way. So what do you call a fine-tune set that expands "F Y I" to ...
#6154: Who Checks the Labels? Weak vs Strong Supervision
Human labels, rule-based labels, or a teacher model writing the exam — what actually makes supervision "strong," and who verifies it?
#6153: Decision Models: Typed Answers in One Forward Pass
Clef, Jev, and OpenAI's Decisions API return typed answers with probabilities in a single forward pass — no generated text to parse.
#6149: 4 Hard Fails in 84 Answers: Testing AI Bias on Israel-Palestine
A 19-prompt, 6-model eval tested whether AI models push back evenly on false claims about Israel and Palestine. One model failed four times.
#6146: Grounding Is Not Solved: DIY vs Gateway vs Native Search
Bolt-on search, gateway search, or native grounding? The plumbing didn't vanish — it just moved to someone else's bill.
#6145: Do LLMs Nod Along With Conspiracy Theories?
A 98-prompt eval asks one question: when a believer brings a conspiracy theory, does the model push back, or help them dig?
#6142: We Gave 4 AI Models the Same Toast Prompt
Four models, one buttered-toast physics question. Three invented studies that never existed — and the oldest model was the most fun.
#6141: Overheard at the Laundrette
A seven-minute laundrette scene about socks hides a real sentence. Can you spot the null cipher before it's revealed?
#6139: Which AI Writes Our Scripts Best? We Tested Five
Five models, three scripts each, one rubric. Sol won on length, Grok on voice — and the runner-up title hides a catch.