AI Evaluation
512 episodes · Page 3 of 22
#6051: How to Find the AI Benchmark You Actually Need
Thousands of benchmarks, no canonical index. How Benchmark Radar and the Lighteval Tasks Explorer help — and where they fall short.
#6050: Codex vs Claude Code: What the Numbers Actually Count
Codex has more users, Claude Code has more revenue — both can be true. The difference is what each number is counting.
#6049: Claude Code by the Numbers: What's Measurable, What's Hype
Anthropic won't publish a user count. Here's what revenue, surveys, and 400K sessions actually tell us about Claude Code's rise.
#6043: Write the Spec Before You Pick the Model
Long context, character voice, cheap at scale — three requirements that fight each other. The document that decides the fight.
#6042: Who Grades the Grader? AI Writing Benchmarks Put to the Test
Maths benchmarks can't see whether a model's prose collapses by chapter three. Five specialist leaderboards try — and our own writer is on them.
#6040: Claude Code Subagents Can Now Spawn Subagents
Claude Code subagents can now spawn their own subagents. Here's where the depth limits come from, and why nobody actually knows where they stop.
#6039: Who's Actually Building Frontier AI Outside the US and China?
Mistral sits 20 points off the frontier. Korea runs a bake-off. The UAE open-sources everything. What's actually driving sovereign AI?
#6038: The Model You Pay For Isn't the Best Deal
Eight Chinese labs, two tiers each, and a pricing spreadsheet that says your default model is mid-pack.
#6037: Gemini 4 Argon: Google's Million-Token Flagship
A close read of Gemini 4 Argon's model card — a million-token window, thinking levels, implicit-only caching, and a lot of blanks.
#6036: GPT-6 Astra: A Million Tokens, No Fine-Tuning
A million-token window, ten built-in tools, a 272K pricing cliff — and no fine-tuning. What GPT-6 Astra actually offers builders.
#6034: Claude Opus 5.5: What Anthropic Actually Shipped
Anthropic's new flagship is cheaper than the model it replaces — and won't let you turn thinking off.
#6032: Blending Datasets vs Merging Weights for Fine-Tuning
Merging weights happens in parameter space. Blending datasets happens in data space — and the messy part is worth it.
#6028: ASR vs Multimodal LLMs: What Actually Won
Dedicated ASR plus a cleanup model is quietly beating end-to-end dictation on the hardware you already own.
#6024: What's Actually Inside the KV Cache?
890 bytes per token: why the KV cache is both what makes LLMs practical and what makes long context expensive.
#6023: How Much Voice Data Is Enough?
Two hours of phone recordings took Hebrew code-switching from 20% to 60%. But when does more data stop mattering?
#6022: Fine-Tuning for Code-Switched Hebrew on 2 Hours of Audio
Two hours of one man's voice beat models trained on 120,000 hours — but only for code-switched Hebrew.
#6009: Vertex AI Is Now the Gemini Agent Platform
Vertex AI just became the Gemini Enterprise Agent Platform. Here's the family tree, the five products, and when to actually move.
#6008: Why Codex Writes Code for a Dead API
Models don't hallucinate your API — they quote real docs from the wrong year. Here's why, and what actually fixes it.
#5994: Hebrew in Latin Letters: A TTS Pipeline Fix
A token classifier that finds Hebrew words written in Latin letters so TTS stops mangling them — built for under $3.
#5989: Twenty Tiny Beats That Make a Podcast Stick
Twenty novel recurring segment pitches — and the test each one has to pass: does it add context, or just noise?
#5982: Teaching a Voice Engine to Say "Yalla" Right
A small token classifier that spots Hebrew words hiding in English sentences — and why finding the word is the easy half.
#5962: How Word Error Rate Lies to Us
WER is the number everyone reports for speech recognition — and the one that agrees with human judgment least. Here's what it actually measures.
#5960: There's No Such Thing as Tokens Per Second
Why one tokens-per-second number can't describe an LLM server — and which metrics actually tell you if it's working.
#5959: What RTF Actually Tells You About Speech Models
Accuracy gets the headlines, but speed decides what ships. How RTF, latency percentiles, and memory shape real ASR deployments.