AI Evaluation
517 episodes · Page 14 of 22
#2186: The AI Persona Fidelity Challenge
Advanced LLMs dominate benchmarks but fail at staying in character—especially when asked to play morally complex or antagonistic roles. What does t...
#2185: Taking AI Agents From Demo to Production
Sixty-two percent of companies are experimenting with AI agents, but only 23% are scaling them—and 40% of projects will be canceled by 2027. The ga...
#2181: When RAG Becomes an Agent
RAG in chatbots is simple retrieval. RAG in agents is a multi-step decision loop. Here's what actually changes.
#2178: How to Actually Evaluate AI Agents
Frontier models score 80% on one agent benchmark and 45% on another. The difference isn't the model—it's contamination, scaffolding, and how the te...
#2175: Let Your AI Argue With Itself
What happens when you let multiple AI personas debate each other instead of asking one model one question? A deep dive into synthetic perspective e...
#2173: Inside MiroFish's Agent Simulation Architecture
MiroFish generates thousands of AI agents with distinct personalities to predict social dynamics. But research reveals a critical flaw: LLM agents ...
#2172: Council of Models: How Karpathy Built AI Peer Review
Andrej Karpathy's llm-council uses anonymized peer review to make language models evaluate each other fairly—but can it really suppress model bias?
#2171: How IQT Labs Built a Wargaming LLM (Then Archived It)
A deep code review of Snowglobe, IQT Labs' open-source LLM wargaming system that ran real national security simulations before being archived. What...
#2165: Strip Your Agent to Bash
The frameworks matter less than you think. What separates a working agent from a failing one is the harness—the orchestration, memory, and tool des...
#2164: Why Bigger Context Windows Don't Fix Attention
Frontier models have million-token context windows, but attention degrades well before you hit the limit. New research reveals why bigger isn't bet...
#2162: When Knowledge Work Stops Being Safe
The knowledge economy promised safety from automation. Then AI arrived. Here's how we got here—and why the disruption this time is different.
#2160: Claude's Latency Profile and SLA Guarantees
Claude is measurably slower than competitors—and Anthropic's SLA promises are even thinner than the latency numbers suggest. What enterprises actua...
#2146: The AI Wargame's Flat Hierarchy Problem
AI wargames treat NGOs and nuclear powers as equals. That's a dangerous flaw for real-world policy planning.
#2144: AI Wargaming: One Model or Many?
Should geopolitical AI simulations use one model or many? We debate the pros and cons of a single-model approach.
#2143: Simulating Geopolitics Under Asymmetric Information
A two-stage AI pipeline predicted a 4% chance the Iran-Israel ceasefire would survive a month, using Monte Carlo simulations and an LLM council.
#2139: AI Wargame Memory: Beyond the Context Window
Why simply extending context windows fails in multi-agent simulations, and how layered memory architectures preserve strategic fidelity.
#2137: Wargaming's Methodology, Not Magic
Most AI wargames are just expensive role-play. Here's the professional methodology they're missing.
#2136: The Brutal Problem of AI Wargame Evaluation
Most AI wargame simulations skip evaluation entirely or rely on token expert reviews. This is the field's biggest credibility problem.
#2135: Is Your AI Wargame Signal or Noise?
Monte Carlo methods promise statistical rigor for AI wargaming, but the line between genuine insight and sampling noise is thinner than you think.
#2133: Engineering Geopolitical Personas: Beyond Caricatures
How to build LLMs that simulate state actors with strategic fidelity, not just surface mimicry.
#2132: The Referee's Dilemma: Epistemic Containment in LLM Simulations
Why do AI war games need a news blackout? We dissect the firewall that keeps LLM actors from cheating with real-world data.
#2131: The CIA Is on GitHub
In-Q-Tel is on GitHub. Explore the IC's strategic investment arm and its use of open-source AI for wargaming.
#2129: Shifting Left on Hallucinations
Stop hoping your AI doesn't lie. We explore the shift to deterministic guardrails, specialized judge models, and the tools making agents reliable.
#2125: Why Agentic Chunking Beats One-Shot Generation
A single prompt can't write a 30-minute script. Here’s the agentic chunking method that fixes coherence.