DPO
Direct Preference Optimization training approach
Episodes
-
#6154: Who Checks the Labels? Weak vs Strong SupervisionHuman labels, rule-based labels, or a teacher model writing the exam — what actually makes supervision "strong," and who verifies it? -
#5922: Teaching a Model Your Voice: SFT vs DPOThree ways to teach a model your writing voice — and the evidence says one of them quietly makes things worse. -
#5412: Editing vs. Note-Taking for AI Fine-TunesHand-editing a model's output gives three training signals at once. Writing notes gives one — and a weaker one at that. -
#4789: How LLMs Actually Know When to StopThree layers of stop signals, from EOS tokens to context limits — and why none of them really "know" when to stop. -
#3283: Fine-Tuning DeepSeek for One PodcastCan a purpose-specific fine-tune fix a model's stubborn writing tics? We explore the practical engineering behind it. -
#2307: Inside Frontier LLM Training: Stages, Costs, and CheckpointsDiscover the multi-stage process of training frontier large language models, from pretraining to post-training, and why checkpoints are the key to ... -
#2177: Skip Fine-Tuning: Shape LLMs With Alignment AloneCan you build a personalized LLM by skipping traditional fine-tuning and using only post-training alignment methods like DPO and GRPO? We break dow... -
#121: Why Your AI Is a Yes-ManEver wonder why AI is so polite? Herman and Corn dive into the mechanics of RLHF and how "niceness" gets baked into modern language models.