HumanEval
Code generation benchmark for LLMs
Episodes
-
#4775: The Evaluator Role No One's Building ForBenchmarks like MMLU are broken. A new role is emerging: the domain-specific AI evaluator. -
#4338: The Hidden Skill Most AI Models Still Fail AtWhy do even top AI models still accidentally paste your questions into the output? A deep dive into a missing benchmark. -
#3596: Why an AI Model Kept Calling Itself Sonnet 4.6When a Chinese model insists it's "Sonnet 4.6," is it theft, sloppy training, or something stranger? -
#3422: How Rival Labs Reverse-Engineer a New AI Model in HoursInside the organized frenzy when a closed-source model drops — and how competitors map its every weakness. -
#3157: Opus 4.8: What Actually Changed Under the HoodAnthropic dropped Opus 4.8 with no fanfare. New training data, faster inference, and smarter refusals — here's what changed. -
#2357: Microsoft's Phi: When Data Quality Beats Model SizeExplore Microsoft AI's Phi family of small language models, designed for edge deployment and high efficiency. -
#2354: Profiling a Ghost ModelA deep dive into Amazon Nova, a mysterious AI model family on Bedrock — and the gaps in what we know. -
#2014: Coding Tools Are Secretly System AgentsThey call it a coding assistant, but real users are treating it like a personal operating system. -
#2008: Needle-in-a-Haystack Testing for LLMsNew AI models claim to be genius-level, but can they actually find a specific fact in a massive document? -
#1364: The Strategic Slowdown: Avoiding the FOMO Architecture TrapLearn how "Integration Scouts" help CTOs cut through AI marketing hype to build modular, future-proof enterprise architectures. -
#1231: The Agentic Shift: 5 Bold AI Predictions for 2026The Poppleberry brothers move past the chatbot era to deliver five high-stakes, falsifiable predictions for the future of autonomous AI agents. -
#798: Beyond the Button: How AI Learns From Your FeedbackEver wonder if your AI feedback actually matters? Discover how ratings shape global models and the privacy tech keeping your data safe. -
#105: Beyond Math Puzzles: The Truth About AI BenchmarksAre AI models getting smarter, or just better at memorizing tests? Herman and Corn dive into the controversial world of 2025 AI benchmarks.