braintrust
LLM evaluation and observability platform
Episodes
-
#5623: When Your AI Cites Sources It Didn't Actually UseYour RAG pipeline cites a source — but did the model actually use it? New interpretability research says you can't tell from the trace. -
#2780: Building Self-Healing Agent PipelinesHow to build an agent that monitors and fixes other agents in production — without the hype. -
#2403: Choosing Your LLM Eval FrameworkAn architectural shootout of four major LLM evaluation harnesses — where each shines and where each breaks down. -
#2254: How to Test an AI Pipeline ChangeWhen you tweak one part of a complex AI agent system, how do you know if it actually improved anything? The answer lies in engineering checkpoints. -
#2185: Taking AI Agents From Demo to ProductionSixty-two percent of companies are experimenting with AI agents, but only 23% are scaling them—and 40% of projects will be canceled by 2027. The ga... -
#2184: The Economics of Running AI AgentsProduction AI agents can cost $500K/month before optimization. Learn model routing, prompt caching, and token budgeting to cut costs 40-85% without... -
#2178: How to Actually Evaluate AI AgentsFrontier models score 80% on one agent benchmark and 45% on another. The difference isn't the model—it's contamination, scaffolding, and how the te... -
#2010: Building Better AI Memory SystemsWe obsess over AI inputs but treat outputs like Snapchat messages. Here's why that's a massive blind spot. -
#1932: How Do You QA a Probabilistic System?LLMs break traditional testing. Here’s the 3-pillar toolkit teams use to catch hallucinations and garbage outputs at scale. -
#1929: From Vibe Checks to Model MetricsWe stopped "vibe-checking" our AI scripts and built a science fair for models. Here's how we grade them. -
#1499: The Black Box Recorder: Why AI Needs an Active ArchiveStop treating AI chats as disposable. Discover why active archiving is now the essential gold standard for enterprise data and compliance.