Fine Tuning
178 episodes
#6502: What a Cross-Entropy Loss of 1.8 Actually Means
A cross-entropy loss of 1.8 nats means a language model is about as uncertain as choosing uniformly among six equally likely tokens.
#6495: Teaching Speech Models to Choose Their Own Audio
Selecting the most uncertain speech recordings often just surfaces silence, clipping, and abandoned half-sentences.
#6494: Where Speech Models Actually Learn
Just three self-attention heads out of twenty in Whisper's decoder trigger over 75 percent of hallucinations on non-speech audio.
#6493: Why Speech Models Forget Other Voices When Tuned on Yours
Adapting Whisper to one person's voice can trigger catastrophic forgetting, stripping away timestamps, segmentation, and accuracy on other speakers.
#6492: What Audio Models Learn Before Fine-Tuning
With just ten minutes of labeled audio on top of unlabeled pretraining, wav2vec 2.0 achieves a 4.8 word error rate on clean speech.
#6484: Why Narrow Speech Tasks Still Need Large Models
NVIDIA experiments show speech models trained from scratch on target domains had nearly double the error rate of fine-tuned pretrained models.
#6463: Why Training Speech Recognition From Scratch Fails
Thirty-two hours of speech provides just three hundred thousand words of text, making from-scratch speech models struggle with basic spelling.
#6458: Why Synthetic Fine-Tuning Data Needs Tooling
Rewriting high-quality data synthetically gives a 3.4-point gain, but running the same process on low-quality data yields just 0.5.
#6457: Where to Learn LLM Training Without Generic Courses
A GPT-2 scale model that took OpenAI 168 hours to train in 2019 can now be replicated on a leaderboard in under two hours.
#6448: Why Fast Model Microtraining Needs an Evaluation Gate
A speech model fine-tuning run can cost twenty-three cents in twenty minutes, but unmonitored updates silently erase older vocabulary.
#6408: Why Speech Model Fine-Tuning Demands Clean Restarts
Fine-tuning only on new vocabulary causes catastrophic forgetting, whereas joint training on old and new data provides the optimal benchmark.
#6368: What Does It Take To Train An AI Model From Scratch?
nanochat trains a BPE tokenizer in Rust in roughly 34 seconds, yet it can quietly disadvantage languages like Korean by nearly half.
#6366: Why Training Recipes Matter More Than Model Size
Altering learning-rate timing alone improved global word error rates by twelve absolute percentage points in speech recognition experiments.
#6365: Teaching Speech Models to Transcribe Bilingual Slang
Fine-tuning Parakeet with 77 minutes of speech cut Hebrew-English code-switching errors nearly in half, but later gains required far more audio.
#6355: Why Model Downgrades Are Rarely About Quantization
A caching bug and a shift to medium reasoning effort once reduced Anthropic internal benchmark scores by about three percent.
#6344: Training Text Models to Know When Not to Edit
In a 2023 WMT Marathi task, an LLM post-editor modified over 90% of sentences but improved only around 20%.
#6332: Why Extra Voice Hours Settle Bilingual Speech
Fine-tuning Parakeet TDT on fifteen hours of bilingual speech cut error rates from 13.7% to 4.9% by resolving contextual variation.
#6317: Choosing Model Precision, Quantization, and Format
Quantizing Qwen 2.5 9B to Q4_K_M boosted generation speed 2.7 times over BF16, while prompt processing rates stayed roughly unchanged.
#6315: Inside China's Mixed-Language AI Models
Domestic Chinese models typically draw sixty to eighty percent of their training data from Chinese, leaving massive room for English and code.
#6312: Choosing Between Modal, RunPod, and Vast.ai
RunPod lists an H100 SXM at $3.99 an hour on Secure Cloud, but its Community marketplace drops it to $2.69.
#6310: Why Model APIs Lack Dedicated Few-Shot Fields
Zou and colleagues found models suffered steep drops at 16,000 tokens on tasks requiring learning across an entire demonstration set.
#6298: Why Dictation Models Steal Other People's Voices
Stock speech models lack target-speaker conditioning, defaulting to transcribing whichever competing voice is loudest or most predictable.
#6288: What Happens Between Model Checkpoints
Extending reinforcement learning on Olmo 3 Think for 21 days yielded a 20-point leap on IFBench without starting fresh pretraining.
#6266: Balancing Speech Training Data With Square Roots
A fine-tuning recording app weighs tasks by the square root of unread sentences so massive datasets don't crowd out smaller prompt sets.