Rope
Rotary position embeddings for context extension
Episodes
-
#5182: DeepSeek V4.1 Flash: 1M Context, 437x Smaller KV CacheDeepSeek V4.1 Flash landed with a 1M-token window and a KV cache 437x smaller than V1. Here's what actually changed — and why the middle of your co... -
#2366: Why LLMs Forget the Middle of Long ConversationsWhy do large language models struggle with the middle of long conversations? Explore the science behind attention dilution and practical fixes. -
#2062: How Transformers Learn Word Order: From Sine Waves to RoPETransformers can’t see word order by default. Here’s how positional encoding fixes that—from sine waves to RoPE and massive context windows.