Series

Benchmarks

Published AI benchmarks, what they measure, and where their methodology breaks down.

22 episodes
  1. 1

    Beyond Math Puzzles: The Truth About AI Benchmarks

    Are AI models getting smarter, or just better at memorizing tests? Herman and Corn dive into the controversial world of 2025 AI benchmarks.

    22:25 · 26 Dec 2025
  2. 2

    How to Spot Gamed Benchmarks in Chinese AI

    Are Chinese AI models actually beating the West, or just gaming the system? Herman and Corn dive into the reality of modern AI benchmarks.

    23:12 · 1 Jan 2026
  3. 3

    Whisper Small Beats Whisper Large in Speed & Accuracy

    A 4GPU benchmark on Ubuntu shows the 1.5B parameter Whisper Large is slower and less accurate than the tiny Whisper Small.

    26:58 · 29 Mar 2026
  4. 4

    The 79% AI Coder: Reasoning vs. Memorization

    AI models now score 79% on coding benchmarks, but a 40-point drop on harder tests reveals the truth.

    23:31 · 31 Mar 2026
  5. 5

    Needle-in-a-Haystack Testing for LLMs

    New AI models claim to be genius-level, but can they actually find a specific fact in a massive document?

    20:49 · 4 Apr 2026
  6. 6

    How to Actually Evaluate AI Agents

    Frontier models score 80% on one agent benchmark and 45% on another. The difference isn't the model—it's contamination, scaffolding, and how the te...

    27:43 · 12 Apr 2026
  7. 7

    How AI Benchmarks Became Broken (And What's Replacing Them)

    The tests we use to measure AI progress are contaminated, saturated, and gamed. Here's what's actually working.

    25:40 · 16 Apr 2026
  8. 8

    When AI Cheats on Cultural Knowledge

    Five benchmarks that reveal how AI systems fail at cultural knowledge — and what their methodologies tell us.

    29:54 · 25 Apr 2026
  9. 9

    LLM Benchmarks Are Full of Noise: Statistical Rigor in AI Evals

    Why most benchmark claims in AI are statistically indefensible — and what to do about it.

    28:03 · 25 Apr 2026
  10. 10

    When Your AI Says No to Everything

    Why LLMs refuse 73% of harmless prompts — and the trade-off between safety and usefulness.

    23:55 · 25 Apr 2026
  11. 11

    When AI Caves: Progressive vs. Regressive Sycophancy

    Why do LLMs agree with you even when you're wrong? We break down the SycEval benchmark and the 78% persistence problem.

    27:44 · 25 Apr 2026
  12. 12

    Why Million-Token Context Windows Can't Handle 3 Reasoning Steps

    Needle-in-a-haystack is dead. Here's what actually measures whether models can think across long documents.

    28:00 · 25 Apr 2026
  13. 13

    How Researchers Actually Measure Censorship in Chinese LLMs

    Beyond headlines: the actual benchmarks, methodologies, and pitfalls in detecting political refusal in Chinese language models.

    30:45 · 25 Apr 2026
  14. 14

    Are Political Bias Benchmarks Actually Measuring Anything?

    Why the Political Compass Test fails, and what researchers are building instead to actually measure model bias.

    26:46 · 25 Apr 2026
  15. 15

    What Tool-Calling Benchmarks Miss About Production Failures

    BFCL, tau-bench, and Nexus each reveal different failure modes. None of them test what actually kills production agents.

    27:59 · 25 Apr 2026
  16. 16

    Hemmingway-1 and the War on Waffle

    A 27B model promises answers without the preamble. Its benchmark is homegrown — and the behavior it targets has a paper trail.

    22:02 · 21 Sept 2026
  17. 17

    Inside Humanity's Last Exam: 2,500 Questions AI Can't Ace

    2,500 questions, ~1,000 experts, and a top score that jumped from 2.7% to 54.8% in 18 months. What is HLE actually measuring?

    32:50 · 14 Oct 2026
  18. 18

    Who Grades the Grader? AI Writing Benchmarks Put to the Test

    Maths benchmarks can't see whether a model's prose collapses by chapter three. Five specialist leaderboards try — and our own writer is on them.

    24:15 · 16 Oct 2026
  19. 19

    How to Find the AI Benchmark You Actually Need

    Thousands of benchmarks, no canonical index. How Benchmark Radar and the Lighteval Tasks Explorer help — and where they fall short.

    35:19 · 17 Oct 2026
  20. 20

    What "Real" Improvement Actually Means in AI Evals

    A p-value isn't the probability your result was luck. Here's what chance actually means in AI benchmarks — and why bold numbers need cross-examinat...

    30:16 · 21 Oct 2026
  21. 21

    Video Models Can't Tell Time: What Motion Benchmarks Reveal

    Frontier models score 48% on tasks where humans hit 98% — the gap is motion, speed, and sequence, not scene description.

    23:02 · 21 Oct 2026
  22. 22

    Ten Weirdest AI Evals Ever Published

    A pelican on a bicycle, a badger for Groundhog Day, and a benchmark where passing is the bad outcome — the strangest AI evals ever run.

    21:13 · 22 Oct 2026