The war for the biggest model is over. For years, we were obsessed with parameter counts, like we were measuring the horsepower of an engine that nobody knew how to drive. We would see these massive numbers—one hundred billion, five hundred billion, one trillion—and we just assumed bigger meant better. But as of March twenty-third, twenty twenty-six, the battlefield has shifted entirely. We have entered what researchers are calling the Deployment Era. Today, it is no longer about who has the most parameters sitting in a cold server room; it is about who can serve the most intelligent tokens at a speed that feels like human thought, or faster. Today's prompt from Daniel is about how massive three-trillion-parameter Mixture of Experts architectures like Grok-three and Grok-four are achieving real-time streaming speeds that frankly should have been impossible for models of that scale just eighteen months ago.
It is a fascinating pivot, Corn. My name is Herman Poppleberry, and I have been obsessively tracking the data from the last few weeks. We are seeing a massive structural change in the industry. As of this month, inference now accounts for sixty-six percent of total artificial intelligence compute spend. Think about that. For the first three years of the boom, almost all the money went into training—into the forge. Now, the money is being spent on the actual usage. We have moved from the era of building the brain to the era of using the brain. The metric that matters now is intelligence density. It is not just about how many parameters you have in the warehouse; it is about how many of them you can actually move through the straw at any given millisecond to give the user an answer. Grok-three is currently averaging about seventy tokens per second, which is a massive jump from the industry average of fifty-nine we saw just last year. When you consider that Grok-three is a three-trillion-parameter beast, that seventy tokens per second is a staggering engineering feat.
Seventy tokens per second for a three-trillion-parameter model sounds like a physics defiance, Herman. If you were running a dense model of that size—where every single parameter had to be activated for every single word generated—the latency would be measured in seconds, or even minutes, not milliseconds. You would be waiting for a response like you are waiting for a letter in the mail. You would type a question, go get a coffee, and maybe the first word would be there when you got back. So, how are they actually cheating the clock here? Because it has to be an architectural trick, right? You can't just throw more electricity at a three-trillion-parameter dense matrix and expect it to stream like a chat bot.
It is less about cheating and more about radical efficiency in how the model activates itself. The core of this magic is the Mixture of Experts, or M-O-E, architecture. When we say Grok-three has three trillion parameters, people imagine a giant wall of three trillion switches all flipping at once for every "hello" or "how are you." But in a sparse M-O-E design, the model only activates a tiny fraction of those parameters for any single token. We are talking about less than five percent. It is like having a massive library with a thousand specialized librarians. If you ask a question about nineteenth-century French poetry, you don't need the librarian who specializes in quantum calculus or the one who knows everything about gardening. You only wake up the poetry experts. This "Conditional Computation" allows the model to have the wisdom of a three-trillion-parameter giant but the footprint and speed of a much smaller model during the actual conversation.
I have seen the term Latent Routing popping up in the research papers lately, specifically with NVIDIA's Nemotron three Super that just dropped on March twelfth. How does that differ from the standard expert routing we saw in the earlier versions of these models? Is it just a faster way to pick the librarian?
It is a much more intelligent way to pick them. Traditional routing was often a bit shallow. A token would hit the router, and based on some very basic features, it would be sent to an expert. This often led to what we call shallow specialization, where experts were not really experts; they were just general-purpose buckets that didn't really know why they were being picked. Latent Routing, which we see heavily utilized in the Grok-four architecture released last July, actually processes the input through several initial dense layers to understand the deep semantic context first. It builds a latent representation of the intent before the router ever makes a decision. By the time the token reaches the specialized layers, the model has a much clearer idea of whether it needs a coder, a poet, or a mathematician. NVIDIA's Nemotron three Super is a great example of this—it is a one-hundred-and-twenty-billion parameter model, but it only activates twelve billion parameters, or ten percent, per pass. Because the routing is so precise, those twelve billion parameters punch way above their weight class, achieving frontier-level reasoning at a tenth of the token cost.
So it is slowing down for a microsecond to understand the "vibe" of the question so it can speed up for the rest of the journey. That makes sense. But even if you pick the right expert, you still have the memory bandwidth problem. We have talked about this before, but the bottleneck in AI is rarely the raw math; it is moving the data from the memory to the processor. I remember we did a deep dive on this back in episode ten eighty-one when we talked about the K-V cache being an invisible memory tax. Is that still the primary hurdle for these three-trillion-parameter models?
The K-V cache is still the taxman, Corn, and he always gets his cut. For those who need a refresher, the Key-Value cache stores the previous parts of the conversation so the model does not have to re-process everything from scratch for every new token. If you are five thousand words into a chat, the model needs to "remember" those five thousand words to predict the next one. For a model with a hundred-and-twenty-eight-thousand-token context window, that cache becomes massive. If you do not manage it, it chokes the memory bandwidth, and your streaming speed drops to a crawl. Grok-three has implemented something called QuantSpec to solve this. It is a four-bit hierarchical quantization of the K-V cache.
Explain that without the jargon, Herman. What is actually happening to the memory of the conversation?
It is essentially compressing the memory of the conversation on the fly, but doing it smartly. Instead of storing every detail of the previous tokens in high precision, it uses four-bit quantization to shrink the footprint. But because it is hierarchical, it keeps the most recent or most relevant tokens—the ones you just typed—at a higher resolution so it doesn't lose the immediate context. It then compresses the older parts of the conversation more aggressively. It is like your own brain; you remember the last sentence I said perfectly, but you only have a "compressed" summary of what I said ten minutes ago. Combined with Grouped-Query Attention, or G-Q-A, which allows multiple "heads" of the model to share the same memory keys, it allows the model to look at the entire context window without having to load a massive amount of data for every single forward pass. This is why you can have a long, complex conversation with Grok and it still streams at seventy tokens per second even when you are ten thousand words deep.
That explains how it handles the history, but what about the actual generation? I have noticed that when I use these newer models, the text does not just appear one letter at a time anymore; it almost feels like it is appearing in small, rhythmic chunks. Is that just a visual trick of the interface to make it feel faster, or is there something happening in the architecture?
That is Multi-Token Prediction, or M-T-P, and it is one of the biggest shifts in the last year. For the longest time, large language models were strictly one-token-at-a-time machines. You predict one word, feed it back in, and predict the next. It is a very linear, very slow process. It is like a writer who has to stop and think for five seconds after every single letter they type. Grok-four and some of the newer DeepSeek variants are now designed to predict two or even four tokens in a single forward pass.
Wait, how can it predict the fourth word before it has even confirmed the second word? That sounds like it is guessing ahead of itself. Isn't that prone to massive errors?
It is exactly a guess, but it is a very calculated one. The architecture has multiple output heads that are trained to look further into the future. It is like a grandmaster chess player who is not just looking at the next move but the next three moves simultaneously. By predicting a block of tokens at once, you effectively multiply your throughput. Even if the fourth token is occasionally wrong, the architectural gain from getting it right most of the time far outweighs the cost of a quick correction. This is a massive part of why we have seen the jump from fifty to seventy tokens per second. The model is essentially saying, "I am ninety percent sure the next four words are 'the quick brown fox,' so I'll just give them to you all at once."
It sounds like the model is constantly sprinting and then checking its own work. Which brings us to speculative decoding. I have heard this described as a draft and verify system. Is Grok using a smaller model to do the heavy lifting for the streaming?
That is exactly the workflow, and it is brilliant. xAI uses what they call Grok-three mini as a drafter. Think of the mini model as a high-speed assistant who is a bit sloppy but very fast. It is much smaller, maybe only a few billion parameters, so it can run incredibly fast on standard hardware. It drafts a sequence of, say, eight potential tokens. Then, the massive three-trillion-parameter Grok-three model—the "Boss"—looks at all eight tokens at once in a single parallel pass. It is much faster for the big model to verify a draft than it is for it to generate those eight tokens one by one. If the big model agrees with the draft, you just got eight tokens for the price of one forward pass. If it disagrees at token number five, it just throws away the rest and starts a new draft from there. If the draft is ninety percent accurate, you are looking at a two-to-three-times increase in throughput.
So the user sees a smooth, high-speed stream because the assistant is constantly throwing out ideas, and the boss is just nodding along until something is wrong. That is a brilliant way to hide the latency of a three-trillion-parameter brain. But does this speed come with a price? I am thinking about the reasoning tax. We have seen these new Think modes where the model pauses to do chain-of-thought before it answers. Does that basically negate all these speed gains?
That is the big tension in the industry right now, Corn. We actually touched on the rise of native reasoning in episode fourteen seventy-three. When you engage Think mode, the model is not just streaming; it is doing test-time compute. It is generating thousands of internal tokens—a private monologue—that the user never sees, just to verify its own logic and explore different paths. For Grok-three, the streaming speed is still high once the answer starts, but the time-to-first-token increases significantly. You might wait three or four seconds of silence while it "thinks." You are trading that immediate gratification for a much higher probability of being correct. The real architectural achievement is that they have decoupled these two things. You can have a fast mode for casual chat and a reasoning mode for complex coding, all using the same underlying M-O-E weights but with different activation patterns.
It feels like a bit of a psychological game. If I am asking for a recipe for lasagna, I want seventy tokens per second. I don't need it to "think" about the existential implications of pasta. But if I am asking for a security audit of a smart contract, I am fine waiting five seconds if it means the model is not hallucinating a vulnerability. But I want to talk about the competition for a second. DeepSeek has been the name on everyone's lips lately because they seemed to have figured out how to do this much cheaper than the American labs. Just a few days ago, on March twentieth, they released DeepSeek-V-three-point-two with something called Sparse Attention. They are claiming to cut long-document inference costs by fifty percent. What did DeepSeek do differently that forced companies like xAI to rethink their entire stack?
DeepSeek-V-three really put the pressure on by proving that you could get frontier-level performance with a much smaller active parameter count. They were only activating about thirty-seven billion parameters out of a six-hundred-and-seventy-one-billion-parameter total. They focused heavily on what they call Multi-Head Latent Attention, which is another way to compress that K-V cache bottleneck we were talking about. By open-sourcing their high-efficiency kernels, they showed the world that a lot of the proprietary speed at places like xAI or Anthropic was not just about having better hardware, but about better math in the attention layers. It essentially embarrassed the closed-source labs into optimizing. Now, we are seeing Grok-three fast variants on Oracle Cloud that are utilizing specialized kernels to hit those seventy tokens per second consistently, largely because they had to keep up with the efficiency benchmarks set by DeepSeek.
It is amazing how much of this comes down to just being clever with how you move bits around. It is not just about throwing more H-one-hundreds at the problem. Speaking of hardware, though, we have to mention the Groq L-P-U integration that NVIDIA announced at G-T-C this month. That is the Language Processing Unit. How does that change the architectural math for these M-O-E models?
The L-P-U is a game changer because it is designed for deterministic latency. Standard G-P-Us are great at many things, but they can be a bit unpredictable with how they handle the branching logic of a Mixture of Experts model. When a router sends a token to an expert, it creates a tiny bit of "jitter" in the timing. The L-P-U is built specifically for the sequential, branching nature of language. When you integrate that with a model like Grok, you get a stream that is not just fast, but perfectly steady. There is no stuttering. It is like the difference between a car that goes fast in bursts and a high-speed train that maintains a constant three hundred miles per hour. This hardware-software co-design—where the chip knows exactly how the M-O-E router works—is where the next ten-times improvement in speed is going to come from.
So if I am a developer looking at this landscape in late March twenty twenty-six, what is the takeaway? Should I be looking at the total parameter count at all when I am picking a model for my app, or is that a dead metric?
It is almost entirely a dead metric for deployment. You should be looking at intelligence density. How much reasoning capability are you getting per millisecond of latency? The cost of inference has dropped from twenty dollars per million tokens in twenty twenty-two to about forty cents per million tokens today. That is a fifty-fold reduction in four years. For a developer, this means you can now build always-on agents that can process massive amounts of data in real-time without breaking the bank. You should prioritize models that use these sparse M-O-E architectures because they give you the best of both worlds: the wisdom of a massive model with the speed of a tiny one. If you are building a real-time voice assistant, you need that seventy tokens per second. If you are building a legal researcher, you look for the "Think" mode.
It is a wild time. I mean, we are sitting here talking about three-trillion-parameter models like they are just another tool in the shed, but the engineering required to make them stream at seventy tokens per second is staggering. It is like trying to make a blue whale swim at the speed of a dolphin. Before we wrap up, I have to ask about the elephant in the room. Or I guess, the giant in the room. Grok-five. The rumors are that it is being trained on the Colossus supercluster with over two hundred thousand G-P-Us—a mix of H-one-hundreds and the new B-two-hundreds. If that is a six-trillion-parameter model, can these architectural tricks even scale that far? Or are we going to hit a wall where the model just becomes too heavy to move, no matter how many experts you have?
There is a theory that we might hit a memory wall, where the sheer size of the weights makes it impossible to load them fast enough, even with sparsity. But I think the research into test-time compute suggests a different path. Instead of just making the model bigger, we might start seeing models that are the same size but just "think" longer. However, knowing xAI and the scale of the Colossus cluster, they will probably try to do both. The real question is whether there is a limit to how much we can compress reasoning. Can you really fit the sum of human knowledge into a four-bit quantized cache and still expect it to understand the nuance of a complex legal contract or a subtle piece of literature? We are pushing the absolute limits of information theory here.
It feels like we are trying to pack the entire ocean into a garden hose and then act surprised when the pressure is high. It is a fascinating technical challenge, and honestly, the fact that we can even have this conversation in real-time with these models is a testament to how far the architecture has come in just the last few months.
It really is. And it is not just about speed for the sake of speed. When a model responds instantly, the friction between human intent and machine execution disappears. That is when it stops feeling like a tool you are operating and starts feeling like an extension of your own mind. That is the real goal of the Inference War.
Well, on that high note, I think we have covered the bases on how these giants are learning to sprint. To summarize for Daniel, it is not just one thing. It is the sparse activation that only wakes up five percent of the brain, it is the speculative decoding that uses a mini-assistant to draft the answers, it is the Multi-Token Prediction that guesses the future, and it is the aggressive compression of the K-V cache that keeps the memory from choking. Put it all together, and you get a three-trillion-parameter beast that can out-type a professional stenographer.
And it's only going to get faster. We are already hearing whispers of sub-ten-cent-per-million-token pricing by the end of the year.
Predicting the future—something we try to do every week on this show, with varying degrees of success.
We are at least more accurate than a four-bit quantized model on its first day of training, I hope.
I wouldn't bet on that, Herman. I've seen your sports brackets from last year.
That is a low blow, Corn. A very low blow. My bracket was a victim of statistical outliers.
Sure it was. Thanks as always to our producer, Hilbert Flumingtop, for keeping the gears turning behind the scenes. And a big thanks to Modal for providing the G-P-U credits that power the generation of this very show. If you want to dive deeper into the technical papers we mentioned today—like the DeepSeek Sparse Attention paper or the Nemotron Latent Routing docs—or if you want to search our entire archive of over fourteen hundred episodes, head over to myweirdprompts dot com. You can find the R-S-S feed there and all the ways to subscribe so you never miss a deep dive.
If you are enjoying the show, a quick review on your podcast app of choice really helps us reach new listeners who are trying to make sense of this fast-moving world. We appreciate every single one of you.
This has been My Weird Prompts. We will be back next time with another prompt from Daniel. Until then, keep your tokens fast and your latency low.
Goodbye, everyone.
