LMSYS Chatbot Arena
Crowdsourced LLM leaderboard
Episodes
-
#4775: The Evaluator Role No One's Building ForBenchmarks like MMLU are broken. A new role is emerging: the domain-specific AI evaluator. -
#3278: How to Get Early AI Model Access as a Solo DeveloperHow a solo developer spending $300/month can get early access to new AI models before the press release. -
#2511: Measuring AI API Latency Through the Black BoxHow to benchmark token throughput and debug slowdowns in closed CLI tools like Claude Code using OpenTelemetry and mitmproxy. -
#2006: How Do You Measure an LLM's "Soul"?Traditional benchmarks can't measure tone or empathy. Here's how to evaluate if an AI model truly "gets it right." -
#1940: Why Google's 31B Model Fits in Your GPUGoogle just dropped Gemma four, and its 31-billion-parameter size is a masterclass in hardware-aware AI design. -
#1929: From Vibe Checks to Model MetricsWe stopped "vibe-checking" our AI scripts and built a science fair for models. Here's how we grade them. -
#1499: The Black Box Recorder: Why AI Needs an Active ArchiveStop treating AI chats as disposable. Discover why active archiving is now the essential gold standard for enterprise data and compliance. -
#584: Will AI Brain Drain Kill the Modern University?Can AI actually do math research? Herman and Corn dive into DeepMind’s Alithia agent and the shift toward "System 2" thinking in AI. -
#130: How to Spot Gamed Benchmarks in Chinese AIAre Chinese AI models actually beating the West, or just gaming the system? Herman and Corn dive into the reality of modern AI benchmarks.