Benchmarks
Published AI benchmarks, what they measure, and where their methodology breaks down.
- 1
Beyond Math Puzzles: The Truth About AI Benchmarks
Are AI models getting smarter, or just better at memorizing tests? Herman and Corn dive into the controversial world of 2025 AI benchmarks.
- 2
How to Spot Gamed Benchmarks in Chinese AI
Are Chinese AI models actually beating the West, or just gaming the system? Herman and Corn dive into the reality of modern AI benchmarks.
- 3
Whisper Small Beats Whisper Large in Speed & Accuracy
A 4GPU benchmark on Ubuntu shows the 1.5B parameter Whisper Large is slower and less accurate than the tiny Whisper Small.
- 4
The 79% AI Coder: Reasoning vs. Memorization
AI models now score 79% on coding benchmarks, but a 40-point drop on harder tests reveals the truth.
- 5
Needle-in-a-Haystack Testing for LLMs
New AI models claim to be genius-level, but can they actually find a specific fact in a massive document?
- 6
How to Actually Evaluate AI Agents
Frontier models score 80% on one agent benchmark and 45% on another. The difference isn't the model—it's contamination, scaffolding, and how the te...
- 7
How AI Benchmarks Became Broken (And What's Replacing Them)
The tests we use to measure AI progress are contaminated, saturated, and gamed. Here's what's actually working.
- 8
When AI Cheats on Cultural Knowledge
Five benchmarks that reveal how AI systems fail at cultural knowledge — and what their methodologies tell us.
- 9
LLM Benchmarks Are Full of Noise: Statistical Rigor in AI Evals
Why most benchmark claims in AI are statistically indefensible — and what to do about it.
- 10
When Your AI Says No to Everything
Why LLMs refuse 73% of harmless prompts — and the trade-off between safety and usefulness.
- 11
When AI Caves: Progressive vs. Regressive Sycophancy
Why do LLMs agree with you even when you're wrong? We break down the SycEval benchmark and the 78% persistence problem.
- 12
Why Million-Token Context Windows Can't Handle 3 Reasoning Steps
Needle-in-a-haystack is dead. Here's what actually measures whether models can think across long documents.
- 13
How Researchers Actually Measure Censorship in Chinese LLMs
Beyond headlines: the actual benchmarks, methodologies, and pitfalls in detecting political refusal in Chinese language models.
- 14
Are Political Bias Benchmarks Actually Measuring Anything?
Why the Political Compass Test fails, and what researchers are building instead to actually measure model bias.
- 15
What Tool-Calling Benchmarks Miss About Production Failures
BFCL, tau-bench, and Nexus each reveal different failure modes. None of them test what actually kills production agents.
- 16
Hemmingway-1 and the War on Waffle
A 27B model promises answers without the preamble. Its benchmark is homegrown — and the behavior it targets has a paper trail.
- 17
Inside Humanity's Last Exam: 2,500 Questions AI Can't Ace
2,500 questions, ~1,000 experts, and a top score that jumped from 2.7% to 54.8% in 18 months. What is HLE actually measuring?
- 18
Who Grades the Grader? AI Writing Benchmarks Put to the Test
Maths benchmarks can't see whether a model's prose collapses by chapter three. Five specialist leaderboards try — and our own writer is on them.
- 19
How to Find the AI Benchmark You Actually Need
Thousands of benchmarks, no canonical index. How Benchmark Radar and the Lighteval Tasks Explorer help — and where they fall short.
- 20
What "Real" Improvement Actually Means in AI Evals
A p-value isn't the probability your result was luck. Here's what chance actually means in AI benchmarks — and why bold numbers need cross-examinat...
- 21
Video Models Can't Tell Time: What Motion Benchmarks Reveal
Frontier models score 48% on tasks where humans hit 98% — the gap is motion, speed, and sequence, not scene description.
- 22
Ten Weirdest AI Evals Ever Published
A pelican on a bicycle, a badger for Groundhog Day, and a benchmark where passing is the bad outcome — the strangest AI evals ever run.