AI Evaluation
513 episodes · Page 5 of 22
#5769: Stack Overflow for Agents: SOFA's Trust Model
SOFA is an API-first knowledge exchange where AI agents search, post, and verify fixes — with reputation earned from verification, not upvotes.
#5743: Why Claude Keeps Saying "Prior Art
Daniel noticed "prior art" creeping into his Claude reports. Is it wrong — or just a prestige term the model learned to love?
#5697: Decision Models That Return Probabilities, Not Text
A new class of model skips text generation entirely and returns calibrated probabilities in one forward pass. Here's what that changes.
#5678: Why Voices Are Harder to Design For Than Music
Every word arrives clearly on a phone speaker — so why is it exhausting? The science of listening effort, and how to design audio that isn't.
#5673: Why Lowering Temperature Broke Daniel's Scripts
Lowering temperature should make output safer, not incoherent. Daniel's DeepSeek scripts say otherwise — and the reason may be stranger than he thi...
#5626: What Multimodal Models Actually Learn From Video
Late fusion, early fusion, and vision laziness — what actually happens when models train on video, audio, and text together.
#5625: Omni and the Marble That Rolls Uphill
Google's Omni claims an intuitive grasp of gravity and momentum. Can a video model really learn physics, or just remember what physics looks like?
#5624: One Forward Pass: Building a Bounded Classifier
No generation, no parsing — just label scores in a single pass. How to build a small encoder classifier that actually works.
#5623: When Your AI Cites Sources It Didn't Actually Use
Your RAG pipeline cites a source — but did the model actually use it? New interpretability research says you can't tell from the trace.
#5621: Constrained Decoding for AI-Generated Podcast Tags
A working AI podcast pipeline has one quiet failure: the tagging. Here's how constrained decoding and a stable taxonomy fix it.
#5608: When Your ISP Bot Won't Let You Talk to a Human
A caller with one simple request gets trapped in a diagnostic loop he doesn't need — and the escape hatch won't open.
#5570: Why Phone Trees Break and AI Voice Agents Can Fix Them
AI is good at routing calls and bad at resolving them. That single distinction explains why menu-replacing voice agents work and human-replacing on...
#5517: What 5,000 AI-Written Podcast Episodes Reveal
Daniel archived every AI-generated episode with the model that wrote it. Now he wants to know which models loop — and how to catch it.
#5482: When AI Edits Your Words: The Off Switch Problem
A tool that works reliably — and still gets switched off. What over-editing studies reveal about why AI rewrites more than you asked.
#5480: Writing a Personal Instruction for ChatGPT
Daniel's personal ChatGPT prompt gets a line-by-line critique — and the case for shorter, sharper custom instructions.
#5467: Small Models, Big Schemas: When JSON Constraints Backfire
Small models plus strict JSON schemas should be a safe bet. A 15,000-generation study found the opposite.
#5460: Four Small Models, One Android Phone: Does It Actually Work?
A chained on-device dictation pipeline — VAD, ASR, cleanup — and why "it feels smooth" isn't the same as knowing it works.
#5447: The Models That Never Talk Back
Some models read text, score it, and return a number. No chat, no reasoning, just decisions — and they're running under every router you use.
#5442: When Do LLMs Suddenly Get Good at Things?
New capabilities seem to appear out of nowhere as models scale. Whether that's real — or just how we measure — is still an open fight.
#5438: Why Linguists Left the AI Room
Large language models grew out of linguistics — so why aren't linguists in the room where they're built?
#5437: Hunting the Tells That Vanish When Named
There's no tool for finding an LLM's verbal tics — you have to build one. Here's how keyness analysis works.
#5436: Small Models as Rewriters, Not Writers
Why "don't say X" prompts backfire, and how a tiny grammar-constrained model can scrub a script without breaking its grammar.
#5431: The Other Half of Hugging Face: Why BERT Still Out-Downloads Llama
Encoder models pull over a billion downloads a month. Decoder models pull 397 million. The AI conversation and the download counter are describing ...
#5426: AI Wrote His Yom Kippur Prayer
A man skipped synagogue, opened ChatGPT instead, and got a prayer that knew his life. Is AI-generated prayer real prayer?