AI Evaluation
520 episodes · Page 13 of 22
#2308: When AI Forecasts Collide: Geopol Model Divergence
Five AI models forecast the Iran-Israel-US crisis — and their disagreements reveal surprising insights about geopolitical reasoning.
#2306: Can LLM Councils Truly Capture Diverse Worldviews?
Exploring whether LLM councils can achieve genuine worldview diversity or if alignment processes erase meaningful differences.
#2288: The Invisible Gatekeeper of Voice Tech
How voice activity detection shapes every step of the voice tech pipeline, and why it’s harder than it seems.
#2282: When Metrics Become the Gate
How do investors cut through the noise in the AI startup surge? We break down the metrics that truly matter—and why MRR alone isn’t enough.
#2279: What a 40% Swing Reveals About Trust
A Jerusalem Post survey shows a 40% shift in Israeli public opinion—what does this tell us about trust in democracies?
#2274: Weekend Projects Gone Wild: Evaluating AI Startup Pitches
From fridge tax agents to guilt-scheduled cron jobs, we evaluate ten AI-driven startup ideas that could exist—but probably shouldn’t.
#2272: The AI Transcription Sweet Spot
Does higher-quality audio make AI transcription worse? New research reveals a surprising "sweet spot" for bitrate, challenging a core assumption of...
#2261: The Gap Between AI Output and Art
We assess if AI can truly invent a Tolkien-level language, write a coherent novel, or author an original screenplay—and where the real gaps in crea...
#2254: How to Test an AI Pipeline Change
When you tweak one part of a complex AI agent system, how do you know if it actually improved anything? The answer lies in engineering checkpoints.
#2250: How Incentives Shape AI Safety Research
Vendor labs, independent research orgs, government agencies—the AI safety field is messier and more diverse than most people realize. A map of wher...
#2249: Building Custom Benchmarks for Agentic Systems
Public benchmarks fail for agentic systems. Learn how to build evaluation frameworks that actually predict production behavior.
#2246: Constitutional AI: Anthropic's Theory of Safe Scaling
How Anthropic's Constitutional AI replaces human raters with AI self-critique guided by explicit principles—and what it assumes about the future of...
#2239: How AI Benchmarks Became Broken (And What's Replacing Them)
The tests we use to measure AI progress are contaminated, saturated, and gamed. Here's what's actually working.
#2233: Who Actually Wants AI to Slow Down?
Daniel argues AI development should slow down for expertise and stability. But who in the industry actually shares this philosophy beyond the obvio...
#2228: Tuning RAG: When Retrieval Helps vs. Hurts
How do you prevent retrieval from suppressing a model's reasoning? We diagnose our own pipeline's four control levers and multi-source fusion strat...
#2224: Why AI Can't Crack the Voynich Manuscript
A fifteenth-century text has defeated cryptanalysts, linguists, and AI models alike. What does its resistance tell us about language, encoding, and...
#2221: Can an AI Have Taste?
Two AI hosts curate 12 podcasts for curious minds—and ask whether an AI can actually have taste in the first place.
#2214: The Three Failure Modes of AI News Systems
When a conflict changes hourly, AI systems built for yesterday's information fail. Here's how to architect pipelines that actually keep up.
#2213: When Ground Truth Moves Hourly
How do you rigorously evaluate whether Tavily or Exa retrieves better results for breaking news? A formal benchmark beats the vibe check.
#2206: What Actually Works in AI Memory
Most AI memory systems are just vector databases with similarity search. We break down what mem0, Zep, and Letta are actually doing—and why benchma...
#2196: The Invisible Workforce Behind AI
Annotation is the invisible foundation of AI—and a $17B industry by 2030. Here's what dataset curators actually need to know about the tools, platf...
#2194: Game Theory for Multi-Agent AI: Design Better, Fail Less
Nash equilibrium, mechanism design, and why your AI agents are playing prisoner's dilemma whether you know it or not.
#2191: Making Multi-Agent AI Actually Work
Research from Google DeepMind, Stanford, and Anthropic reveals most multi-agent systems waste tokens and amplify errors. Single agents with better ...
#2190: Simulating Extreme Decisions With LLMs
LLMs fail at the exact problem wargaming was built to solve—simulating irrational, extreme decision-makers. A new study reveals why.