LM Evaluation Harness
Framework for property-based LLM evaluation
Episodes
-
#6051: How to Find the AI Benchmark You Actually NeedThousands of benchmarks, no canonical index. How Benchmark Radar and the Lighteval Tasks Explorer help — and where they fall short. -
#4444: Testing the Unpredictable: QA for Agentic AIHow QA adapts when your AI system gives different answers to the same question every time. -
#2005: Beyond Vibes: The Hard Science of LLM EvaluationRunning the same LLM on different GPUs can produce different results. Here’s why that happens and how to test for it. -
#1762: Testing AI Truthfulness: Beyond VibesStop trusting confident AI. We explore the formal science of testing LLMs for hallucinations and knowledge cutoffs.