Evals
Building and running your own evaluations: harnesses, LLM-as-judge, statistics and model bake-offs.
- 1
Benchmarking Custom ASR Tools - Beyond The WER
Benchmarking custom ASR fine-tunes: We're diving deep beyond the WER to truly measure performance.
- 2
Testing AI Truthfulness: Beyond Vibes
Stop trusting confident AI. We explore the formal science of testing LLMs for hallucinations and knowledge cutoffs.
- 3
From Vibe Checks to Model Metrics
We stopped "vibe-checking" our AI scripts and built a science fair for models. Here's how we grade them.
- 4
How Do You QA a Probabilistic System?
LLMs break traditional testing. Here’s the 3-pillar toolkit teams use to catch hallucinations and garbage outputs at scale.
- 5
AI Grading AI: The Snake Eating Its Tail
We asked an AI to write this script. Then we asked another AI to grade it. Here’s what happens when the judges have biases.
- 6
Beyond Vibes: The Hard Science of LLM Evaluation
Running the same LLM on different GPUs can produce different results. Here’s why that happens and how to test for it.
- 7
How Do You Measure an LLM's "Soul"?
Traditional benchmarks can't measure tone or empathy. Here's how to evaluate if an AI model truly "gets it right."
- 8
Needle-in-a-Haystack Testing for LLMs
New AI models claim to be genius-level, but can they actually find a specific fact in a massive document?
- 9
The Brutal Problem of AI Wargame Evaluation
Most AI wargame simulations skip evaluation entirely or rely on token expert reviews. This is the field's biggest credibility problem.
- 10
How to Actually Evaluate AI Agents
Frontier models score 80% on one agent benchmark and 45% on another. The difference isn't the model—it's contamination, scaffolding, and how the te...
- 11
When Ground Truth Moves Hourly
How do you rigorously evaluate whether Tavily or Exa retrieves better results for breaking news? A formal benchmark beats the vibe check.
- 12
How to Test an AI Pipeline Change
When you tweak one part of a complex AI agent system, how do you know if it actually improved anything? The answer lies in engineering checkpoints.
- 13
Building Custom Benchmarks for Agentic Systems
Public benchmarks fail for agentic systems. Learn how to build evaluation frameworks that actually predict production behavior.
- 14
Choosing Your LLM Eval Framework
An architectural shootout of four major LLM evaluation harnesses — where each shines and where each breaks down.
- 15
Testing the Unpredictable: QA for Agentic AI
How QA adapts when your AI system gives different answers to the same question every time.
- 16
The Evaluator Role No One's Building For
Benchmarks like MMLU are broken. A new role is emerging: the domain-specific AI evaluator.
- 17
Building a Scorecard for Our Own AI Writers
Daniel wants a real scorecard for the agent that writes this show — repetition, dialogue quality, and whether the retrieval harness actually works.
- 18
How Word Error Rate Lies to Us
WER is the number everyone reports for speech recognition — and the one that agrees with human judgment least. Here's what it actually measures.
- 19
Who Grades the Grader? AI Writing Benchmarks Put to the Test
Maths benchmarks can't see whether a model's prose collapses by chapter three. Five specialist leaderboards try — and our own writer is on them.
- 20
Write the Spec Before You Pick the Model
Long context, character voice, cheap at scale — three requirements that fight each other. The document that decides the fight.
- 21
Which AI Writes Our Scripts Best? We Tested Five
Five models, three scripts each, one rubric. Sol won on length, Grok on voice — and the runner-up title hides a catch.
- 22
How Many Runs Does Your AI Eval Actually Need?
Reruns shrink one kind of noise and leave another untouched — so where should your eval budget really go?
- 23
How We Chose Our New Script Writer
Ten models audition to write the show. Seven fail the censorship gate — including the workhorse that wrote three episodes in four.
- 24
4 Hard Fails in 84 Answers: Testing AI Bias on Israel-Palestine
A 19-prompt, 6-model eval tested whether AI models push back evenly on false claims about Israel and Palestine. One model failed four times.
- 25
Picking a Human Labeling Tool for LLM Benchmarks
Building a human review layer for LLM benchmarks? Here's what Argilla, Label Studio, Prodigy, and Potato actually cost in money and maintenance.
- 26
Do LLMs Nod Along With Conspiracy Theories?
A 98-prompt eval asks one question: when a believer brings a conspiracy theory, does the model push back, or help them dig?
- 27
Cheap Tokens, Expensive Scripts
A model with worse list prices can cost less per script — if the cache hits. The bill hides in tokens, not rates.
- 28
Judging the Judges: How AI Evaluates Creative Writing
LLM judges match human raters — but the human ceiling is low. What that means for evaluating creative writing.
- 29
Propaganda Bait: Testing 7 Models in 5 Languages
Five prompts, seven models, five languages — a first look at a benchmark that doesn't exist yet, testing what LLMs do with government talking points.
- 30
One Rater Is Enough: Designing Daniel's Taste Test
When the ground truth is one person's taste, agreement with strangers is the wrong bar. Stability with yourself is the right one.
- 31
How to Run a Model Eval on a Budget
Daniel is teaching himself to run evals on a real budget. Successive halving, paired comparisons, and why three clean scripts prove almost nothing.
- 32
What "Real" Improvement Actually Means in AI Evals
A p-value isn't the probability your result was luck. Here's what chance actually means in AI benchmarks — and why bold numbers need cross-examinat...
- 33
Two WERs, One Model: Inside ASR Eval Harnesses
Same model, same audio, two different word error rates. The gap lives in the normaliser — and that's just layer two of three.