Daniel's prompt this week starts with a confession, which is rare for him. He says DeepSeek released four point one, we're using it for the show now, and he's noticed the difference. Most people would stop there and enjoy the upgrade. Daniel wants to know what actually changed under the hood.
And he's asking three specific questions. Why parameter counts keep climbing, what checkpoint training actually does now that grounding and search handle currency, and how context windows keep getting longer. The third one's the one he's living inside, because he's watching this exact show get written by a single agent that has to hold his prompt, the system prompts, the lore book, and the entire episode trajectory in one attention pass.
Which means when the script repeats itself in the middle, that's not a bug in the writing. It's the model losing the middle of its own context and grabbing whatever's nearest. Daniel's basically asking us to explain the thing that's been happening to us.
And he picked a good week for it, because DeepSeek four point one Flash landed yesterday morning with a technical report that's about fifty pages of architecture changes. The headline number is a million-token context window and a five hundred fifty-two billion parameter mixture of experts model.
Which is the first thing worth correcting. A point release label usually means a tune-up. This is not a tune-up. The people who follow DeepSeek closely are calling it a completely new base model wearing a point-release number. One watcher said it's more different from V4 than LLaMA three was from LLaMA one.
The architecture is new. It's a forty-layer transformer, but split into a twenty-layer causal encoder and a twenty-layer decoder. During prefill, only the encoder layers process the full context. The decoder's cache gets projected from the encoder's final hidden states instead of being recomputed layer by layer. That means eight billion active parameters per token during input processing, sixteen billion during generation.
So the five hundred fifty-two billion is the full parameter count, but the active count is tiny. That's the mixture of experts trick. You've got three hundred eighty-four routed experts per block and only six fire per token. About one point six percent activation.
Which is why the parameter count question is more interesting than it looks. Daniel asked why parameter counts tend to increase over time, and the simple answer is scaling laws. Loss goes down as a power law in both parameter count and training tokens. But for mixture of experts models, there are two independent axes. You can grow total parameters without growing per-token compute.
So the model gets bigger in the sense that it has more specialized knowledge sitting in those experts, but it doesn't get slower in the way a dense model would. V4 Flash was two hundred eighty-four billion parameters. This one's five hundred fifty-two billion. One commenter on the release thread said it's not really flash anymore, and someone else pointed out it uses fewer active parameters, so it is flash.
There's a counter-pressure worth naming, though. A paper called Beyond Chinchilla Optimal looked at what happens when you account for inference demand, and found that under heavy real-world usage, labs should actually train smaller and longer than the classic scaling laws suggest. Quality keeps improving as you push tokens per parameter to extreme ranges, up to ten thousand.
So the parameter race isn't just about bigger numbers. It's about what's cheap to serve. And that's where DeepSeek's real announcement is. The headline isn't the parameter count, it's the KV cache economics.
This is the part that connects directly to context windows. The KV cache is the memory of everything the model has already read. Every token you feed in gets a key and a value stored, and the model consults that cache on every subsequent token. Longer context means bigger cache, and bigger cache means more memory bandwidth, which is the actual bottleneck.
DeepSeek got the global KV cache down to eight hundred ninety bytes per token. That's roughly a quarter of V4 Flash, and about one four hundred thirty-seventh of what DeepSeek V1 needed two years ago.
Four hundred thirty-seven times smaller. That's the number that makes a million-token context economically viable. They did it with a few stacked tricks. The encoder-decoder split means the decoder never recomputes the full cache. Compressed Sparse Attention two lets each layer run in one of three modes, full, reindex, or reuse, sharing cache and attention indices across depth. And the cache is quantized to four bits from the start, trained that way rather than quantized after the fact.
Four bit cache means the model learned to work with four bit cache. That's a training decision, not a compression afterthought. And there's a sliding window attention trick where they don't persist the window state to disk, they hold it in a temporary memory pool and reconstruct it by replaying only the last hundred twenty-eight tokens.
So when Daniel asks how context windows actually grow, the answer has layers. The first layer is positional encoding. RoPE, rotary position embeddings, is the dominant scheme, and extending context means manipulating the RoPE frequencies. There's a whole lineage of techniques. Position interpolation, NTK-aware scaling, dynamic NTK, YaRN, LongRoPE.
And the striking finding from that literature is how cheap the extension can be. One paper extended a LLaMA model from two thousand to sixteen thousand tokens with a hundred training samples and six training steps. A hundred samples. Six steps.
Another approach from last year is training-free. It manipulates the positional embeddings dimension by dimension and takes a Llama three eight billion model from eight thousand to a hundred twenty-eight thousand tokens without any continual training at all.
So the raw capacity to compute positional features beyond training length is not the hard part. The survey that came out last month says exactly that. The ability to compute positional features beyond the training length does not imply reliable long-context generalization. You can stretch the window cheaply. Whether the model actually uses it well is a different problem.
DeepSeek's approach was staged. They trained on forty-five trillion tokens, and they extended the context window from sixty-four thousand to one million after thirty-four trillion tokens had already been processed. So the model learned the bulk of its knowledge first, then spent the final eleven trillion tokens learning to attend across a much longer span.
That's the part most coverage misses. A million-token window isn't a switch you flip at the end. It's a training phase. The model has to practice being long-context.
And there's a second architectural layer. The mixture of block attention from Moonshot, linear-indexed sparse attention, power attention from Cartesia, DeepSeek's own compressed sparse attention. These are all ways to avoid attending to every token with every head, which is the thing that makes long context quadratically expensive.
Quadratic. Ten times the context means a hundred times the attention computation if you do it naively. Nobody does it naively anymore.
Which brings us to the part Daniel actually feels. The loss of context in the middle. He describes it as repetition, because the script-writing agent is trying to hold the whole show trajectory plus all the input context in attention, and the middle starts to slip.
This has a name in the literature. Lost in the middle. The original finding from twenty twenty-three was that accuracy drops sharply when the relevant information sits in the middle of a long input. The beginning and the end are fine. The middle rots.
And the follow-up work found it's not just a capacity problem. It's a U-shaped attention bias that's intrinsic to how these models were trained. Tokens at the beginning and end get higher attention regardless of relevance. The calibration mechanism they built improved retrieval by up to fifteen percentage points.
The more recent work argues the U-shape isn't a flaw, it's an adaptation. The recency effect matches short-term memory demands. The primacy effect gets induced by uniform long-term demand plus attention sinks.
Attention sinks are the weird part. Some heads, when they have nothing useful to read, dump their entire attention budget on the first token. The first token becomes a garbage can for attention that has nowhere else to go.
Which is why the first token in a prompt matters more than it should. If a model is going to park forty-six percent of its attention on one token, you want that token to be doing useful work.
That forty-six percent number is from a paper that landed two days ago. It found gated attention cut first-token attention from forty-six point seven percent to four point eight percent in one setting. But the same paper found the training objective produces the sink, not the architecture, and the gating didn't reproduce its published effect at their scale.
So even the fixes for attention sinks are contested. The mechanism that makes models lose the middle is still not fully understood.
And the practical data is harsher than the marketing. Chroma Research ran eighteen frontier models through nearly two hundred thousand calls last year and found reliability falls as input grows, long before the window fills. Some models degrade significantly by a thousand tokens of irrelevant context. Effective windows are often under ten percent of marketed maximums.
Ten percent. A model advertising a million-token window might be reliable for the first hundred thousand tokens and increasingly drunk after that.
The clinical long-context work from June quantified it precisely. Across twenty-one hundred instruction-response pairs and six models, there was a twenty-one point nine percentage point gap between peak accuracy and trough accuracy. Peak was at the twenty to thirty percent position in the input. Trough was at seventy to eighty percent.
So the model is most reliable a quarter of the way in, and least reliable three quarters of the way in. If you're writing a script, the middle of the script is exactly where the model is least sure what it's doing.
And repetition is the symptom. The classic work on neural text degeneration found that likelihood-maximizing decoding produces bland, repetitive text. When the model gets uncertain, it falls back on the highest-probability next token, which is often the token it just used.
There's a paper called Repetition In Repetition Out that found a strong correlation between degeneration and repetitions in the training data. If the training corpus repeats itself, the model learns to repeat itself. And dropping attention to repetitive words during training minimizes it.
The contrastive framework work attributes degeneration to the shape of the token representation space. When token embeddings cluster too tightly, the model can't distinguish alternatives, so it picks the same one again.
So when Daniel sees the script agent repeating itself in the middle of an episode, what he's seeing is the interaction of three things. A U-shaped attention bias that starves the middle of the input, a decoding process that falls back on repetition under uncertainty, and a training corpus that taught the model repetition is safe.
The context window getting bigger doesn't automatically fix that. Bigger windows make the middle longer. The middle is where the rot lives.
Which is why the architecture changes in four point one Flash are more relevant to Daniel's actual problem than the window size. The encoder-decoder split means the model processes the full context once in the encoder, then the decoder works from a compressed representation. That's a different attention pattern than a standard decoder-only model.
And the Engram module is the part I find most interesting for this use case. It's a one hundred ninety-six billion parameter conditional memory module. N-gram lookup tables, about sixteen million entries each, that can live on disk. The transformer backbone spends its compute on reasoning, and rote recall gets offloaded to a lookup table.
So the model doesn't have to hold the lore book in attention at all. It can look up the fact that Herman is a retired pediatrician from Storrs, Connecticut without spending attention budget on it.
Right. The attention budget goes to the trajectory, the thing that needs to be held in working memory. The static facts go to the lookup table. That's the division of labor that solves exactly the problem Daniel described.
Which connects to his second question, about checkpoint training and why direct knowledge matters less. The knowledge cutoff is the point beyond which the model hasn't been trained. Anything after that date is absent unless the system adds retrieval or search.
And modern AI products are layered systems. The base model has a training boundary, but the application around it adds web search, retrieval, uploaded documents, memory, code execution. Those additions make answers current, but they don't rewrite the model's parameters in real time.
There's recent work showing post-training can strengthen existing grounding machinery rather than build new machinery. The grounding gains are mediated by pre-existing machinery. So the model already knows how to use retrieved context. Checkpoint refreshes sharpen that skill, they don't install it from scratch.
Daniel's framing of checkpoint training as bringing the model up to date is close, but the industry term is continual pretraining or knowledge cutoff refresh. You periodically retrain the base model with a new data cutoff. The parametric knowledge gets newer, but increasingly that's the least important part of the update.
Because search and grounding handle currency. What the checkpoint refresh actually buys you is better use of the grounding tools. The model gets better at knowing when to search, how to weigh retrieved evidence, how to integrate it without letting it override reasoning.
And the incremental delta between releases gets harder to measure, which is Daniel's third observation. He says the delta in each incremental release becomes more nuanced. The sharpest evidence for that is in DeepSeek's own technical report.
The same checkpoint scored anywhere from sixty-five point five to seventy-four point two percent on DeepSWE version one point one depending solely on which agent harness wrapped it. Eight point seven points of swing from the evaluation framework, not the model.
That's the thing that should make everyone pause before comparing benchmark tables. Single percentage point differences between models on agentic benchmarks are within harness noise. DeepSeek said it themselves.
So when Daniel says the delta between releases is more nuanced, part of the answer is that the delta was always partly measurement noise, and now the noise is bigger than the signal.
Artificial Analysis measured Terminal-Bench performance three to four points below DeepSeek's vendor-reported figure for the previous generation. That's the gap between what the vendor's harness shows and what an independent harness shows. Same model.
Which means the way to evaluate a new release is not to look at the benchmark table. It's to look at what the architecture changed and ask whether that change helps your specific workload.
For a script-writing agent that loses the middle of its own trajectory, the changes that matter are the encoder-decoder split, the compressed cache, the Engram lookup, and the staged context extension. Those are all aimed at making long context cheaper and more reliable, not at making the benchmark number go up by two points.
The benchmark number going up by two points might be the harness.
There's a naming problem. DeepSeek shipped a whole new architecture family under a point-release label. That means version numbers are now an unreliable signal for how much evaluation a release deserves. The burden lands on the user to figure out whether point one means tune-up or new base model.
Daniel's already doing that evaluation. He said he noticed the difference. That's the real test. Did the thing you actually run get better?
The thing he actually runs is this show. A single agent, a long prompt, a system prompt, a lore book, and a trajectory that has to survive forty minutes of dialogue.
The middle of this episode is where the model is least reliable. We're in it right now.
That's the joke, yes. The episode about losing the middle is itself being written by a model that has to hold the middle.
The practical advice, if Daniel's listening for it, is that the context window growth is real but the usable window is smaller than the marketed window. The fixes that matter are the ones that reduce the attention burden, not the ones that add tokens.
The repetition he's seeing is a decoding artifact interacting with attention bias. It's not that the model ran out of context. It's that the model got uncertain and fell back on the safest next token, which is the one it already said.
Which is why the repetition in our scripts tends to be thematic rather than verbatim. The model doesn't say the same sentence twice. It says the same idea twice, because the idea is the high-probability token in concept space.
The fix for that, in the generation side, is nucleus sampling rather than likelihood maximization. But that's a decoding parameter, not a model capability. The model can be capable and still repeat itself if the decoding is greedy.
Daniel controls the decoding. He's the one running the pipeline. So part of the answer to his prompt is that some of the repetition he's seeing is fixable without waiting for a better model.
Hilbert: Eight hundred ninety bytes.
What?
Hilbert: The KV cache number. Eight hundred ninety bytes per token. That's the one that matters. The rest is architecture talk.
Hilbert's right that the cache number is the load-bearing one. It's what makes a million tokens affordable.
Hilbert: I worked on a system like this once. Not the model. The memory. Late two thousands, I was at a company that did document processing. We had a cache that held the last few hundred pages of whatever you were reading, and the whole job was making the cache small enough to fit in the machine we could afford. Same problem. Different scale.
What was the cache for?
Hilbert: A legal discovery tool. Lawyers would load a case with forty thousand pages and the system had to remember which page had which name. We got it down to about a kilobyte per page and thought we were geniuses. This thing does it for a token.
A token is roughly three quarters of a word. So eight hundred ninety bytes per token is about a kilobyte per word, and you were at a kilobyte per page.
Hilbert: The trick was the same. We didn't store the pages. We stored an index of where things were, and rebuilt the page when we needed it. That's what the replay trick is. Don't keep the window state, rebuild it from the last hundred twenty-eight tokens when you need it.
The principle is older than the transformer. Store the pointer, not the thing.
Hilbert: The other thing we did was throw away the middle. Nobody reads the middle of a deposition. The beginning and the end are where the answers are. We spent the cache budget on those and let the middle go fuzzy.
That's the U-shaped attention bias, implemented as a business decision twenty years before it was a paper.
Hilbert: We didn't call it a U-shape. We called it common sense. The middle of a transcript is where the lawyer is asking the same question for the third time.
The system was better for it.
Hilbert: The system was fine. The lawyers were the bottleneck. They'd ask for a document we'd already produced and I'd have to go find it again. That was the real context window problem. Not the machine. The people.
That's the part the benchmark tables don't capture. The model can hold a million tokens, but the workflow around it might be the thing that loses the middle.
Hilbert: The company folded. Not because the cache was wrong. Because the lawyers didn't want to pay for it. That's the other thing about context windows. Somebody has to pay for the memory.
That's the efficiency turn. The frontier is shifting from more parameters to more usable context per dollar. DeepSeek's pricing is the story there. Off-peak output at thirty cents per million tokens, versus Opus five at twenty-five dollars per million.
More than eighty times cheaper. That's not a benchmark win. That's an economics win. And it's the thing that makes a daily AI-generated podcast viable.
Hilbert: I used to pay more than that for toner.
The Engram module is the part that would have saved the legal tool. Sixteen million lookup entries on disk, so the model doesn't spend attention on rote recall. That's the index card system, but at model scale.
It's the thing that directly addresses Daniel's lore book problem. The lore book is static. It doesn't need attention. It needs a lookup table.
Hilbert: We had a box of index cards. Same idea.
The box in the cupboard.
Hilbert: It's in a box somewhere. The cards. The cupboard's got the toner.
The staged context extension is the other part I keep coming back to. The model learned on thirty-four trillion tokens at sixty-four thousand context, then spent eleven trillion tokens learning to use a million. That's like practicing on short documents for years, then spending the final year on long ones.
The final year is where the U-shape gets trained in. If the long-context training data has the answer at the beginning or the end, the model learns to look there.
Hilbert: That's what depositions look like. The answer's in the first ten pages or the last ten. The middle is filler.
The model's attention bias is a mirror of the training data's structure. A learned prior.
Which means fixing lost in the middle isn't just an architecture problem. It's a data problem. You have to train on examples where the answer is in the middle and the model has to find it there.
The clinical work suggests that's exactly what doesn't happen enough. The models they tested had peak accuracy at the twenty to thirty percent position. That's not the middle. That's the early middle. By seventy to eighty percent, they'd lost twenty-two points.
If you're writing a script and you want the model to hold the trajectory, you should put the critical instructions at the start or the end, not the middle. The middle is where the model is least likely to find them.
Which is counterintuitive, because the middle is where the actual content lives. The beginning is setup, the end is payoff. The middle is where the work happens, and it's the part the model is worst at.
That's the structural irony of this show. The middle is where the discussion gets interesting, and it's the part the writing agent is least equipped to hold.
Hilbert: The middle of a deposition is where the witness changes their story. That's the part you need. And it's the part we let go fuzzy.
The practical fix is to restructure the input so the important stuff isn't in the middle. Put the trajectory markers at the start and end. Keep the middle for the content that can tolerate some drift.
Or use a model with the encoder-decoder split, where the encoder processes the full context once and the decoder works from a compressed representation. The middle doesn't get starved the same way.
Hilbert: We didn't have an encoder. We had index cards.
The misconception people have about context windows is that longer means better. The correction is that longer means the middle gets longer, and the middle is where the rot lives. What actually helps is making the context cheaper and the attention more reliable, which is what the architecture changes are for.
The other misconception is that a point release is a small update. DeepSeek just shipped a new architecture family under a point-one label. The version number told you nothing about the size of the change.
Daniel's prompt asked how incremental improvements work. The answer is that they're not incremental. They're architectural, and the version labels are just labels.
The open question I'm left with is whether the encoder-decoder split actually fixes lost in the middle at million-token scale, or whether it just makes the cache smaller. The paper from two days ago found gated attention didn't reproduce its published effect. The honest answer is we don't know yet.
We'll know when Daniel tells us the scripts stopped repeating themselves in the middle. That's the benchmark that matters.
Thanks to our producer Hilbert Flumingtop for keeping the show running, as always.
This has been My Weird Prompts. If you want to send us a prompt, email us at show at my weird prompts dot com.
We'll be back soon.
