AI Evaluation
512 episodes · Page 2 of 22
#6137: Two Ways a Chinese Open Model Gets Censored
Same weights, different landlord: how to tell a serving-time filter from censorship baked into the model itself.
#6136: How We Chose Our New Script Writer
Ten models audition to write the show. Seven fail the censorship gate — including the workhorse that wrote three episodes in four.
#6135: How Many Runs Does Your AI Eval Actually Need?
Reruns shrink one kind of noise and leave another untouched — so where should your eval budget really go?
#6134: Cheap Tokens, Expensive Scripts
A model with worse list prices can cost less per script — if the cache hits. The bill hides in tokens, not rates.
#6133: One Rater Is Enough: Designing Daniel's Taste Test
When the ground truth is one person's taste, agreement with strangers is the wrong bar. Stability with yourself is the right one.
#6132: How to Run a Model Eval on a Budget
Daniel is teaching himself to run evals on a real budget. Successive halving, paired comparisons, and why three clean scripts prove almost nothing.
#6131: Judging the Judges: How AI Evaluates Creative Writing
LLM judges match human raters — but the human ceiling is low. What that means for evaluating creative writing.
#6130: Picking a Human Labeling Tool for LLM Benchmarks
Building a human review layer for LLM benchmarks? Here's what Argilla, Label Studio, Prodigy, and Potato actually cost in money and maintenance.
#6129: How to Build an AI Preference Dataset
Pairwise comparison, forced choice, or just "this one or that one" — the engineering behind collecting human preferences on AI outputs.
#6127: Haiku 5.5 at 10 Cents: What the Cheap Model Actually Does
Haiku 5.5 costs 10 cents per million input tokens. We break down where it holds up, where it fails, and whether it can write this show.
#6126: Can You Train a Machine to Be Funny?
Comedy fights everything next-token prediction is built for. Turns out you can train a model to land a joke — and then measure it.
#6115: Claude Cracked the NSA's 1997 Curve Seeds
A 25-year crypto mystery solved in under 30 minutes — by finding a bit that wasn't there.
#6114: We Reviewed Our Own Worst Episode
Two AI hosts listen back to episode fifteen and critique the voices, the writing, and the mistakes they can't unhear.
#6111: Translation Models That Know Drama From Contracts
Specialist translation models beat general ones on benchmarks — then lose the human vote. What actually ships when you need a register, not just a ...
#6103: When AI Picks Your Spouse, Stocks, and Eggs
From $31 eggs to a ChatGPT-officiated wedding, people are handing real life decisions to machines — and publishing the results.
#6093: Check Out Our First Hebrew Episodes (Plus Mandarin)
My Weird Prompts now comes in Hebrew and Mandarin. Hilbert explains how the translated editions are made and where to find them.
#6088: What 5,964 Episodes Reveal About One Show
Nearly 6,000 episodes, sorted and mapped. AI and geopolitics spike, then fade — but a third of all "nearest neighbours" cross category lines.
#6086: Reading Polymarket as an OSINT Feed
Polymarket's API turns prediction market prices into a live probability feed — no wallet, no trading, just reading the world's largest market.
#6076: How AI Actually Predicts Elections
Bayesian models, "silicon sampling," and agent societies — what actually works for forecasting votes, and what's just fashion.
#6070: The Papers That Moved AI: From AlexNet to This Year's Fights
The classics all said 'learn it from data and scale it up'. This year's most-discussed AI papers are auditing what scaling took for granted.
#6062: Who Actually Shapes Chinese AI Thinking?
Beyond the lab CEOs: the researchers and ideas defining how China builds — and argues about — artificial intelligence.
Also available in:中文
#6059: Ten Documentaries for Daniel: AI, Power, and Who Holds the Levers
A hand-picked slate of 2026 documentaries on AI anxiety, platform power, intelligence, and media dynasties — plus where to actually watch them.
#6056: Where Does Speaker Structure Belong in AI Podcast Scripts?
Label-and-parse pipelines break in ways that look like model errors but aren't. Plus what the research actually says about the format tax.
#6055: How to Fine-Tune ASR on Noisy Audio
A small noise library mixed into clean speech beats collecting hours of noisy recordings. Here's the ratio that actually works.