AI Evaluation
520 episodes · Page 12 of 22
#2411: Are Political Bias Benchmarks Actually Measuring Anything?
Why the Political Compass Test fails, and what researchers are building instead to actually measure model bias.
#2410: How Researchers Actually Measure Censorship in Chinese LLMs
Beyond headlines: the actual benchmarks, methodologies, and pitfalls in detecting political refusal in Chinese language models.
#2409: When AI Cheats on Cultural Knowledge
Five benchmarks that reveal how AI systems fail at cultural knowledge — and what their methodologies tell us.
#2406: Why Million-Token Context Windows Can't Handle 3 Reasoning Steps
Needle-in-a-haystack is dead. Here's what actually measures whether models can think across long documents.
#2405: LLM Benchmarks Are Full of Noise: Statistical Rigor in AI Evals
Why most benchmark claims in AI are statistically indefensible — and what to do about it.
#2404: What Tool-Calling Benchmarks Miss About Production Failures
BFCL, tau-bench, and Nexus each reveal different failure modes. None of them test what actually kills production agents.
#2403: Choosing Your LLM Eval Framework
An architectural shootout of four major LLM evaluation harnesses — where each shines and where each breaks down.
#2396: Predicting War: The Science of Geopolitical Forecasting
How do experts predict wars before they happen? Explore the high-stakes world of geopolitical forecasting, from Cold War models to AI-driven simula...
#2377: Is Geopolitical Neutrality a Sustainable AI Strategy?
How DeepSeek carved a niche with efficiency, neutrality, and innovative dialogue handling — and what it means for AI's future.
#2357: Microsoft's Phi: When Data Quality Beats Model Size
Explore Microsoft AI's Phi family of small language models, designed for edge deployment and high efficiency.
#2354: Profiling a Ghost Model
A deep dive into Amazon Nova, a mysterious AI model family on Bedrock — and the gaps in what we know.
#2353: Evaluating Enterprise AI: Palmyra X5
Explore Palmyra X5, Writer’s flagship AI model designed for enterprise workloads, featuring a million-token context window and agentic capabilities.
#2352: The Structured Output Gap in Vision APIs
How do object detection APIs like Gemini, AWS Rekognition, and YOLO compare for automated annotation workflows?
#2348: Diffusion Models Take on Text Generation
Explore Inception Labs’ Mercury 2, a groundbreaking diffusion-based language model that rethinks text generation and reasoning.
#2342: How Python Ate Wall Street
Over 80% of equity trades are now executed algorithmically. How did Python libraries quietly democratize quant finance?
#2340: How AI Models Track a Ship Seizure’s Ripple Effects
When the US seized an Iranian cargo ship, three AI models reshuffled their predictions overnight. Here’s what they saw—and where they disagreed.
#2337: When Diarization Fails Silently
Discover how PyAnnote and other tools tackle the critical task of identifying "who spoke when" in audio—and why it’s harder than it sounds.
#2334: How AI Flattens Your Voice in Emails
Why AI-generated emails feel impersonal and how to reclaim your authentic voice in professional communication.
#2328: How to Spot a Real AI Hackathon
Discover how to identify worthwhile AI hackathons, build meaningful connections, and maximize your impact in virtual communities.
#2315: How to Update AI Models Without Starting Over
Exploring the challenge of updating AI models with new knowledge without costly full retraining.
#2314: One Model or Three? Inside Claude's Architecture
What makes Claude’s Haiku, Sonnet, and Opus different? Discover how architecture shapes their unique strengths and weaknesses.
#2313: When AI Optimizes the Wrong Thing
Discover how AI systems learn to optimize for rewards—and why they sometimes get it dangerously wrong.
#2311: Danish AI: Bridging the Localization Gap
How does AI handle Danish? Explore the challenges and progress in making AI tools work for small-language populations.
#2309: Blind Ranking AI's Best Podcast Scripts
How do 15 AI models handle controversial podcast prompts? We rank their scripts blind and reveal the surprising winners.