AI Evaluation
516 episodes · Page 16 of 22
#1932: How Do You QA a Probabilistic System?
LLMs break traditional testing. Here’s the 3-pillar toolkit teams use to catch hallucinations and garbage outputs at scale.
#1929: From Vibe Checks to Model Metrics
We stopped "vibe-checking" our AI scripts and built a science fair for models. Here's how we grade them.
#1913: AI Context Windows Are Junk Drawers
Stop paying for old messages. Here's how to keep your AI sessions clean and on-topic.
#1907: Why We Still Fine-Tune in 2026
Despite million-token context windows, fine-tuning remains essential. Here’s why behavior, not just facts, matters.
#1905: How VCs Verify AI Startups Without Stealing Code
From the "No-NDA Paradox" to AWS bill forensics, here’s how investors separate real AI from Raspberry Pis in fancy cases.
#1893: AI as a Strategic Adversary for Startups
Can AI stress-test your startup idea before investors do? We explore using AI as a strategic adversary to find blind spots.
#1851: AI Toasters and Poetic Gym Coaches: Why We’re Drowning in Useless AI
From smart toasters that need Wi-Fi to email rewriters that sound like corporate robots, here are the most baffling AI features we’ve seen.
#1839: AI's Data Kitchen: From Hoovering to Fine-Tuning
We go behind the curtain of the AI data pipeline, revealing the messy, multi-billion-dollar war over data curation.
#1838: Tuning Search Without Losing Your Mind
Modern search bars are AI decision engines. Here's how small teams can tune fuzzy matching, semantic search, and reranking without breaking everyth...
#1831: The 79% AI Coder: Reasoning vs. Memorization
AI models now score 79% on coding benchmarks, but a 40-point drop on harder tests reveals the truth.
#1827: Can AI Rewrite a Human Career Path?
We fed our producer's resume to Gemini 1.5 Flash to see if an AI can plot a better career path than he has.
#1819: Claude's 55-Day Personality Transplant
Anthropic leaked 55 days of system prompt updates. See exactly how they rewired Claude's personality, safety rules, and self-awareness.
#1818: Inside Claude's Constitution: A System Prompt Deep Dive
We analyzed Claude Opus 4.6's full public system prompt to uncover its hidden rules for safety, product behavior, and refusal logic.
#1813: Why Hard Tech Is Moving to Jerusalem
Jerusalem's tech sector grew 40% in 2024-25, outpacing Tel Aviv. Discover why this ancient city is now Israel's hub for AI, biotech, and cybersecur...
#1812: When AI Gets a Truth Tether to the Talmud
Sefaria's new MCP server connects AI directly to 2,700 years of Jewish texts, transforming how scholars and curious learners study ancient literature.
#1810: Why Your TTS Sounds Great in English, Terrible Everywhere Else
English AI voices are polished, but global languages hit a wall. Here's why text-to-speech breaks down for Hebrew, Hindi, and beyond.
#1794: RAG Is Cheaper Than You Think (Until It’s Not)
From a $1 embedding bill to a $10k/month vector database bill, here’s the real math behind RAG in 2026.
#1786: When AI Supervisors Fire AI Workers
A new "Agent-in-the-Loop" framework lets AI models manage and terminate other AI agents in real-time.
#1774: The Internal Heat Shield: Telling Hard Truths in DevRel
DevRel isn't just swag and conferences—it's the critical feedback loop keeping developers loyal in an AI-driven world.
#1763: Backend Grunt Work Is Dead. What Now?
AI agents now write 80% of boilerplate code, but the real backend engineering challenges remain.
#1762: Testing AI Truthfulness: Beyond Vibes
Stop trusting confident AI. We explore the formal science of testing LLMs for hallucinations and knowledge cutoffs.
#1756: The Ferrari in the Mud: Prestige Flops
We count down the five worst serious movies of the last five years, starting with a sci-fi disaster that wasted $80 million.
#1752: Whisper Small Beats Whisper Large in Speed & Accuracy
A 4GPU benchmark on Ubuntu shows the 1.5B parameter Whisper Large is slower and less accurate than the tiny Whisper Small.
#1735: The Agentic Stone Age: A Retrospective
We revisit the chaotic rise of BabyAGI and AutoGPT, exploring why their promise of total autonomy led to spectacular failure.