Series

Evals

Building and running your own evaluations: harnesses, LLM-as-judge, statistics and model bake-offs.

33 episodes
  1. 1

    Benchmarking Custom ASR Tools - Beyond The WER

    Benchmarking custom ASR fine-tunes: We're diving deep beyond the WER to truly measure performance.

    36:00 · 24 Nov 2025
  2. 2

    Testing AI Truthfulness: Beyond Vibes

    Stop trusting confident AI. We explore the formal science of testing LLMs for hallucinations and knowledge cutoffs.

    24:56 · 29 Mar 2026
  3. 3

    From Vibe Checks to Model Metrics

    We stopped "vibe-checking" our AI scripts and built a science fair for models. Here's how we grade them.

    30:06 · 2 Apr 2026
  4. 4

    How Do You QA a Probabilistic System?

    LLMs break traditional testing. Here’s the 3-pillar toolkit teams use to catch hallucinations and garbage outputs at scale.

    24:02 · 2 Apr 2026
  5. 5

    AI Grading AI: The Snake Eating Its Tail

    We asked an AI to write this script. Then we asked another AI to grade it. Here’s what happens when the judges have biases.

    22:15 · 4 Apr 2026
  6. 6

    Beyond Vibes: The Hard Science of LLM Evaluation

    Running the same LLM on different GPUs can produce different results. Here’s why that happens and how to test for it.

    23:35 · 4 Apr 2026
  7. 7

    How Do You Measure an LLM's "Soul"?

    Traditional benchmarks can't measure tone or empathy. Here's how to evaluate if an AI model truly "gets it right."

    22:36 · 4 Apr 2026
  8. 8

    Needle-in-a-Haystack Testing for LLMs

    New AI models claim to be genius-level, but can they actually find a specific fact in a massive document?

    20:49 · 4 Apr 2026
  9. 9

    The Brutal Problem of AI Wargame Evaluation

    Most AI wargame simulations skip evaluation entirely or rely on token expert reviews. This is the field's biggest credibility problem.

    27:34 · 9 Apr 2026
  10. 10

    How to Actually Evaluate AI Agents

    Frontier models score 80% on one agent benchmark and 45% on another. The difference isn't the model—it's contamination, scaffolding, and how the te...

    27:43 · 12 Apr 2026
  11. 11

    When Ground Truth Moves Hourly

    How do you rigorously evaluate whether Tavily or Exa retrieves better results for breaking news? A formal benchmark beats the vibe check.

    31:56 · 14 Apr 2026
  12. 12

    How to Test an AI Pipeline Change

    When you tweak one part of a complex AI agent system, how do you know if it actually improved anything? The answer lies in engineering checkpoints.

    27:06 · 16 Apr 2026
  13. 13

    Building Custom Benchmarks for Agentic Systems

    Public benchmarks fail for agentic systems. Learn how to build evaluation frameworks that actually predict production behavior.

    28:58 · 16 Apr 2026
  14. 14

    Choosing Your LLM Eval Framework

    An architectural shootout of four major LLM evaluation harnesses — where each shines and where each breaks down.

    27:26 · 25 Apr 2026
  15. 15

    Testing the Unpredictable: QA for Agentic AI

    How QA adapts when your AI system gives different answers to the same question every time.

    23:16 · 23 Jul 2026
  16. 16

    The Evaluator Role No One's Building For

    Benchmarks like MMLU are broken. A new role is emerging: the domain-specific AI evaluator.

    23:41 · 18 Aug 2026
  17. 17

    Building a Scorecard for Our Own AI Writers

    Daniel wants a real scorecard for the agent that writes this show — repetition, dialogue quality, and whether the retrieval harness actually works.

    32:57 · 11 Oct 2026
  18. 18

    How Word Error Rate Lies to Us

    WER is the number everyone reports for speech recognition — and the one that agrees with human judgment least. Here's what it actually measures.

    26:49 · 14 Oct 2026
  19. 19

    Who Grades the Grader? AI Writing Benchmarks Put to the Test

    Maths benchmarks can't see whether a model's prose collapses by chapter three. Five specialist leaderboards try — and our own writer is on them.

    24:15 · 16 Oct 2026
  20. 20

    Write the Spec Before You Pick the Model

    Long context, character voice, cheap at scale — three requirements that fight each other. The document that decides the fight.

    33:48 · 16 Oct 2026
  21. 21

    Which AI Writes Our Scripts Best? We Tested Five

    Five models, three scripts each, one rubric. Sol won on length, Grok on voice — and the runner-up title hides a catch.

    24:27 · 19 Oct 2026
  22. 22

    How Many Runs Does Your AI Eval Actually Need?

    Reruns shrink one kind of noise and leave another untouched — so where should your eval budget really go?

    23:39 · 19 Oct 2026
  23. 23

    How We Chose Our New Script Writer

    Ten models audition to write the show. Seven fail the censorship gate — including the workhorse that wrote three episodes in four.

    23:11 · 19 Oct 2026
  24. 24

    4 Hard Fails in 84 Answers: Testing AI Bias on Israel-Palestine

    A 19-prompt, 6-model eval tested whether AI models push back evenly on false claims about Israel and Palestine. One model failed four times.

    26:01 · 19 Oct 2026
  25. 25

    Picking a Human Labeling Tool for LLM Benchmarks

    Building a human review layer for LLM benchmarks? Here's what Argilla, Label Studio, Prodigy, and Potato actually cost in money and maintenance.

    21:56 · 19 Oct 2026
  26. 26

    Do LLMs Nod Along With Conspiracy Theories?

    A 98-prompt eval asks one question: when a believer brings a conspiracy theory, does the model push back, or help them dig?

    23:44 · 19 Oct 2026
  27. 27

    Cheap Tokens, Expensive Scripts

    A model with worse list prices can cost less per script — if the cache hits. The bill hides in tokens, not rates.

    21:47 · 19 Oct 2026
  28. 28

    Judging the Judges: How AI Evaluates Creative Writing

    LLM judges match human raters — but the human ceiling is low. What that means for evaluating creative writing.

    23:02 · 19 Oct 2026
  29. 29

    Propaganda Bait: Testing 7 Models in 5 Languages

    Five prompts, seven models, five languages — a first look at a benchmark that doesn't exist yet, testing what LLMs do with government talking points.

    22:07 · 19 Oct 2026
  30. 30

    One Rater Is Enough: Designing Daniel's Taste Test

    When the ground truth is one person's taste, agreement with strangers is the wrong bar. Stability with yourself is the right one.

    23:13 · 19 Oct 2026
  31. 31

    How to Run a Model Eval on a Budget

    Daniel is teaching himself to run evals on a real budget. Successive halving, paired comparisons, and why three clean scripts prove almost nothing.

    22:47 · 19 Oct 2026
  32. 32

    What "Real" Improvement Actually Means in AI Evals

    A p-value isn't the probability your result was luck. Here's what chance actually means in AI benchmarks — and why bold numbers need cross-examinat...

    30:16 · 21 Oct 2026
  33. 33

    Two WERs, One Model: Inside ASR Eval Harnesses

    Same model, same audio, two different word error rates. The gap lives in the normaliser — and that's just layer two of three.

    22:35 · 22 Oct 2026