Training Data
322 episodes
#6503: Why Tokenizer Vocabulary Size Is a Hardware Choice
Inference-optimal vocabulary size shifts sixteen-fold depending on serving batch size, from 32,000 at batch one to 524,000 at batch sixty-four.
#6502: What a Cross-Entropy Loss of 1.8 Actually Means
A cross-entropy loss of 1.8 nats means a language model is about as uncertain as choosing uniformly among six equally likely tokens.
#6498: Why You Must Read AI Traces Before Writing Evals
Reading just thirty real interaction traces by hand reveals failure modes that generic automated evaluation metrics consistently obscure.
#6495: Teaching Speech Models to Choose Their Own Audio
Selecting the most uncertain speech recordings often just surfaces silence, clipping, and abandoned half-sentences.
#6494: Where Speech Models Actually Learn
Just three self-attention heads out of twenty in Whisper's decoder trigger over 75 percent of hallucinations on non-speech audio.
#6493: Why Speech Models Forget Other Voices When Tuned on Yours
Adapting Whisper to one person's voice can trigger catastrophic forgetting, stripping away timestamps, segmentation, and accuracy on other speakers.
#6466: Conferences Rule Machine Learning, Not Journals
At NeurIPS 2025, over 21,000 papers required nearly 22,000 reviewers, yet fewer than half a percent earned oral presentations.
#6464: Teaching Speech Recognizers to Stay Quiet
Masking just three self-attention heads in Whisper large-v3 cuts hallucinations on non-speech audio by over eighty percent.
#6463: Why Training Speech Recognition From Scratch Fails
Thirty-two hours of speech provides just three hundred thousand words of text, making from-scratch speech models struggle with basic spelling.
#6458: Why Synthetic Fine-Tuning Data Needs Tooling
Rewriting high-quality data synthetically gives a 3.4-point gain, but running the same process on low-quality data yields just 0.5.
#6457: Where to Learn LLM Training Without Generic Courses
A GPT-2 scale model that took OpenAI 168 hours to train in 2019 can now be replicated on a leaderboard in under two hours.
#6456: The Unsung Builders of Modern Speech AI
OpenAI's Whisper lists famous co-founders among its six authors, yet shipped with no launch event, demo day, or canonical interview.
#6453: Who Actually Won the Open Weight Race
With over four billion downloads, the most downloaded open-weight model in history is an embedding tool, not a chatbot.
#6444: Teaching Speech Models to Collapse Spelled Words
Fine-tuning formatting rules takes surprisingly little data: a Johns Hopkins study found truecasing accuracy jumps with only fifty documents.
#6398: Why Named Entity Linking Fails After the Span Is Found
Zero-shot large language models still lag behind supervised systems on named entity recognition benchmarks by over eleven F1 points.
#6388: Why Swipe Decks Make Better Training Data
A Tinder-style swipe deck removes comparison to force isolated choices, turning rapid binary labels into active learning training signals.
#6375: Why Modern AI Rests on Statistics
When polygenic risk scores derived from European data are applied globally, prediction accuracy drops to 25 percent in African populations.
#6368: What Does It Take To Train An AI Model From Scratch?
nanochat trains a BPE tokenizer in Rust in roughly 34 seconds, yet it can quietly disadvantage languages like Korean by nearly half.
#6366: Why Training Recipes Matter More Than Model Size
Altering learning-rate timing alone improved global word error rates by twelve absolute percentage points in speech recognition experiments.
#6365: Teaching Speech Models to Transcribe Bilingual Slang
Fine-tuning Parakeet with 77 minutes of speech cut Hebrew-English code-switching errors nearly in half, but later gains required far more audio.
#6362: Where Mathematicians Actually Work and How AI Fits In
A surge of over 700 automated AI proof manuscripts hit the mathematics community in October, testing how researchers verify claims.
#6344: Training Text Models to Know When Not to Edit
In a 2023 WMT Marathi task, an LLM post-editor modified over 90% of sentences but improved only around 20%.
#6333: Filtering arXiv in the Age of Generative AI
arXiv submission volume doubled in two years, prompting a new rule capping each submitter at two papers per month.
#6332: Why Extra Voice Hours Settle Bilingual Speech
Fine-tuning Parakeet TDT on fifteen hours of bilingual speech cut error rates from 13.7% to 4.9% by resolving contextual variation.