We spend all this time talking about the latest large language models, the newest chatbots, and these massive emergent behaviors in artificial intelligence, but we almost never talk about the actual language that all of it is written in. I am not talking about Python or even C plus plus. I am talking about the invisible layer that sits between the math and the silicon. Today's prompt from Daniel is about C-U-D-A, or Compute Unified Device Architecture, and the massive software moat that NVIDIA has built around graphics processing unit programming. Daniel wants us to dig into why this proprietary language is the oxygen of the AI industry and why competitors like AMD and Intel are finding it so difficult to break that lock in.
This is a foundational topic, Corn. I am Herman Poppleberry, and I have been looking forward to this because it really is the secret sauce. Most people see NVIDIA as a hardware company that makes these powerful H-one-hundred or B-two-hundred chips, but if you listen to Jensen Huang, he will tell you that NVIDIA is actually a software company. C-U-D-A launched all the way back in two thousand six. That is a twenty year head start. Think about what was happening in two thousand six. We were still a year away from the first iPhone, and Jensen was already betting the entire company on the idea that graphics chips could be used for general purpose parallel computing.
That bet is paying off to the tune of an eighty-eight percent profit margin today. For the uninitiated, or even for the tech literate who have not touched a kernel, what is the moat exactly? If I am a developer, I am usually writing in PyTorch or TensorFlow. I am not necessarily staring at raw C-U-D-A code all day. Why does it matter what is happening underneath the hood?
That is a common misconception. People think because they use a high level framework like PyTorch, the underlying hardware is interchangeable. But those frameworks are essentially giant wrappers for over four hundred specialized libraries that NVIDIA has optimized over two decades. We are talking about C-U-D-N-N for deep neural networks, C-U-B-L-A-S for linear algebra, and N-C-C-L, which is the NVIDIA Collective Communications Library. When you want to train a model across ten thousand G-P-Us, N-C-C-L is what handles the lightning fast communication between those chips. If you switch to an AMD chip, you are not just switching a piece of silicon; you are losing access to that entire ecosystem of highly tuned, battle tested libraries that have been squeezed for every last drop of performance.
It is like trying to switch from an iPhone to a different phone, but realizing that every single app you use was built specifically for iOS and does not even have an equivalent on the other side. Let's get into the metal for a second. Daniel's prompt mentions how this works at the hardware level. Why is writing for a G-P-U so much more difficult than writing for a standard C-P-U, or central processing unit?
It requires a completely different mental model. A standard processor is like a few genius professors who can solve any complex problem you give them, one at a time. A graphics processing unit is like ten thousand elementary school students who can only do simple addition, but they can all do it at the exact same time. To make that work, C-U-D-A introduces the concept of a kernel, which is a function that runs thousands of times in parallel. But you have to manage how those threads are grouped. In NVIDIA's architecture, threads run in groups of thirty-two called warps.
I love that term, warp. It sounds like science fiction, but the reality of it is actually quite punishing for a programmer, isn't it?
It really is. There is this classic problem called warp divergence. Imagine those thirty-two threads are all supposed to be following the same instruction. But then your code has an if-else statement. If half the threads take the if path and the other half take the else path, they cannot run at the same time anymore. The hardware has to disable half the threads, run the first path, then disable the other half and run the second path. You effectively just cut your performance in half because your code branched. Learning how to write code that avoids those pitfalls while managing shared memory and global memory coalescing is an art form. NVIDIA has twenty years of documentation, debuggers like C-U-D-A-G-D-B, and profilers like N-sight that help engineers solve these specific, hair pulling problems.
And that is where the competitor's struggle begins. If I am AMD and I come out with the M-I-three-hundred-X, which is a beast of a chip, I am still asking developers to walk away from twenty years of specialized tools. AMD has been pushing R-O-C-M, which stands for Radeon Open Compute, as their answer. They even have this tool called H-I-P, the Heterogeneous-Compute Interface for Portability. The idea is that you can take your C-U-D-A code, run it through a script, and it spits out code that runs on AMD. Is it actually that simple?
It is somewhere in the middle. H-I-P is actually surprisingly good for what it is. It is a translation layer that maps C-U-D-A functions to R-O-C-M functions. For a lot of standard deep learning workloads, you can get a codebase ported over with relatively modest effort. In fact, PyTorch now officially supports R-O-C-M on Linux as a first class citizen. You go to the PyTorch install page, and R-O-C-M is right there next to C-U-D-A. That is a massive milestone. But the issue is not just the initial port; it is the optimization. If your translated code runs twenty percent slower because the R-O-C-M version of a specific library isn't as optimized as NVIDIA's, you are still going to buy the NVIDIA chip. Time is money when you are spending millions of dollars on compute.
That leads us to the hardware reality of twenty twenty-six. NVIDIA still holds about eighty-six percent of the data center revenue. That is down slightly from the ninety percent we saw a couple of years ago, but it is still a dominant position. However, AMD is finally starting to land some punches on the hardware side. The M-I-three-hundred-X has one hundred ninety-two gigabytes of high bandwidth memory, while the NVIDIA H-one-hundred only has eighty. That is a two point four times advantage in memory capacity. In an era where models are getting bigger and bigger, surely that memory advantage starts to outweigh the software inconvenience?
That is exactly why AMD is finding a foothold in inference. If you are just running a model, not training it from scratch, memory is king. More memory means you can fit larger models on a single chip or use larger batch sizes, which brings your cost per token down. We are seeing AMD's M-I-three-fifty-five-X deliver about thirty percent faster inference than NVIDIA's B-two-hundred on something like Llama three point one four hundred five billion. When you factor in that AMD chips are generally cheaper, the tokens per dollar calculation starts looking very favorable. This is why we are seeing hyperscalers like Meta, Microsoft, and Google starting to build out these massive AMD clusters. It is not just about performance; it is about having a second source so they have leverage when they sit down to negotiate with Jensen Huang.
It is the ultimate insurance policy. Even if they prefer NVIDIA, they need to prove they can leave. But what about the training side? That still feels like NVIDIA's fortress. We saw the M-L-Perf Training version five point zero results recently, and while AMD's M-I-three-twenty-five-X actually beat the NVIDIA H-two-hundred by about eight percent in some tests, those are very specific, highly tuned benchmarks. Does that translate to the average engineering team trying to train a custom model?
For most teams, the answer is still no. Training is where you run into the most edge cases. It is where you are more likely to need a custom kernel or a specific communication pattern that might not be perfectly optimized in R-O-C-M yet. And we cannot ignore the hardware-software feedback loop. NVIDIA's engineers talk to the PyTorch team and the OpenAI team every single day. When a new paper comes out with a new architectural trick, NVIDIA has an optimized kernel for it within weeks. AMD is getting faster, but they are still playing catch up. However, there is a third player in this game that might be the real giant killer, and it is not a hardware company. It is OpenAI's Triton.
I was waiting for you to bring up Triton. For those who haven't been following the compiler wars, Triton is a language and compiler that allows you to write highly efficient G-P-U kernels using a syntax that looks a lot more like Python than C plus plus. The genius of it is that it is hardware agnostic.
Precisely. Triton acts as an abstraction layer. You write your kernel in Triton, and it can compile down to NVIDIA's P-T-X code or AMD's specific instruction sets. This is the escape valve for the entire industry. If OpenAI and Meta start writing all their core kernels in Triton instead of raw C-U-D-A, then the hardware becomes a commodity. You just pick whichever chip gives you the best performance per watt or per dollar that day. This is a massive threat to NVIDIA's moat because it moves the lock in from the proprietary C-U-D-A language to an open source language that NVIDIA doesn't control.
It is a compelling pivot. And it explains why NVIDIA recently announced that twenty-six billion dollar bet on open source AI. It feels like they realize the C-U-D-A wall is eventually going to be climbed, so they are trying to build a new moat further out. They want to be the ones providing the foundational models, the agentic frameworks, and the entire stack of AI services. If they can't lock you in at the compiler level, they will lock you in at the application level.
It is a classic move. But let's look at why Intel is struggling so much in this space. They have less than one percent of the discrete AI accelerator market right now. They have their one-A-P-I and S-Y-C-L initiatives, which are based on open standards, but it just hasn't gained any real traction. Their current chip, the Gaudi three, is decent for inference, but their next big hope, the Crescent Island G-P-U, isn't even sampling until later this year or early twenty twenty-seven. By that point, NVIDIA will likely have announced the successor to the Blackwell architecture. Intel is essentially two generations behind in a race where a generation lasts eighteen months.
It is a tough spot to be in when you used to own ninety-nine percent of the server market. It shows that you cannot just throw money at the problem if you do not have the developer mindshare. Speaking of mindshare, AMD's R-O-C-M seven point two release was a big deal for developers, specifically because of the Windows support. For a long time, if you wanted to do serious AI work on AMD, you had to be on Linux. Bringing that to Windows means a whole new class of developers and researchers can start experimenting with R-O-C-M on their local machines.
That is a huge part of the chicken and egg problem. If a researcher can't run the code on their workstation, they aren't going to write their paper using that framework. And if the papers aren't written for that framework, the enterprise doesn't adopt it. By bringing R-O-C-M to Windows and making it a first class citizen in PyTorch, AMD is finally starting to address the developer experience. They are trying to make it so that the switch is invisible. If you can just change one line of code from device equals C-U-D-A to device equals R-O-C-M and everything just works, the moat disappears. We are not there yet, but R-O-C-M seven point zero and seven point two have closed that gap significantly. In many benchmarks, we are seeing R-O-C-M within ten to fifteen percent of C-U-D-A performance, and in some inference workloads, it is actually faster because of that memory bandwidth advantage.
So if you are a software engineer listening to this, and you want to future proof your career, what is the move? Do you still double down on learning the intricacies of C-U-D-A, or do you start looking at these abstraction layers like Triton or H-I-P?
I think the smart move is to understand the underlying principles of G-P-U architecture, because those don't change regardless of the language. You still need to understand memory coalescing, thread hierarchies, and how to avoid warp divergence. If you understand those concepts, you can move between C-U-D-A, R-O-C-M, and Triton with relative ease. But if I had to pick one language to master right now for the long term, it would be Triton. It is where the industry is heading because nobody wants to be beholden to a single vendor forever.
It is the same story we have seen in every other layer of the tech stack. We moved from proprietary mainframes to x-eight-six, from proprietary Unix to Linux, and from proprietary cloud A-P-Is to containers and Kubernetes. The moat is always a temporary state of affairs in technology, even if that temporary state lasts twenty years. But eighty-eight percent margins for twenty years is one hell of a run for NVIDIA.
And they aren't going down without a fight. The Blackwell architecture and its successors are integrating more and more specialized hardware for things like transformer engines and F-P-four precision. They are making the hardware so specialized for the current state of AI that even if you have a great compiler, you might still want the NVIDIA chip because it has a specific circuit designed to speed up the exact math your model is doing. It is a moving target.
Let's talk about the bifurcated market idea you mentioned earlier. Do you really see a world where training stays on NVIDIA and inference moves to a commodity market? Because that would be a massive shift in the economics of AI. If the expensive part, the training, is the only place NVIDIA can keep their margins, but the high volume part, the inference, goes to AMD and Intel and custom silicon, that changes the valuation of these companies significantly.
I think that is the most likely outcome. Training is a research and development activity where speed and flexibility are everything. You want the best tools, the best support, and the most stable environment because you are burning millions of dollars a day. You don't want to be debugging a R-O-C-M compiler error when you are trying to train Llama four. But inference is a production activity where you are looking at cost per user. If an AMD cluster can serve the same model for twenty percent less, the C-F-O, or Chief Financial Officer, is going to mandate the switch. We are already seeing this with the hyperscalers building their own custom silicon, like Google's T-P-U or Amazon's Trainium and Inferentia. They are basically saying, we will use NVIDIA to find the next big thing, but once we know what it is, we are going to run it on our own cheaper hardware.
It makes total sense. It is the classic innovation versus commodity cycle. But it is wild to me that we are in twenty twenty-six and we are still talking about a software moat that started in two thousand six. It really highlights how much of a visionary Jensen Huang was. He wasn't just building a faster chip; he was building a language that would eventually become the foundation for a new era of computing.
And he did it by being incredibly generous to the academic community. For a decade, NVIDIA was giving away G-P-Us to every university and researcher who wanted them. They were building the curriculum. If you were a computer science student in twenty fifteen and you wanted to learn parallel programming, you learned C-U-D-A. That is how you build a moat. You don't build it with lawyers; you build it with an entire generation of engineers who don't know how to work any other way.
Well, that generation is growing up, and some of them are the ones building Triton and R-O-C-M now. It is going to be a fascinating few years. I think the big takeaway for me is that the invisible language of AI is finally becoming visible because the stakes have become so high. When it was just a niche thing for researchers, nobody cared about the proprietary lock in. Now that it is the backbone of the global economy, everyone is looking for a way out.
And that is exactly why this is the most important conversation in AI that most people aren't having. We talk about the what of AI all day, but the how is where the power and the money actually live. If you want to understand where the industry is going, don't look at the chatbots; look at the compilers.
I think that is a perfect place to start wrapping this up. We have covered the twenty year head start of C-U-D-A, the technical hurdles like warp divergence that make G-P-U programming so difficult, and the emerging escape valves like OpenAI's Triton and AMD's R-O-C-M seven point two. It feels like we are at a genuine turning point where the NVIDIA or nothing era is evolving into a more complex, multi-vendor reality.
It is the maturation of the industry. We are moving from the wild west phase to the industrial phase. And in the industrial phase, efficiency and second sourcing are the only things that matter.
Before we head out, I want to give a big thanks to our producer, Hilbert Flumingtop, for keeping the gears turning behind the scenes. And of course, a huge thank you to Modal for providing the G-P-U credits that power this show and allow us to do this deep dive research.
If you found this technical deep dive useful, please do us a favor and leave a review on your podcast app. It really does help other people find the show and keeps us motivated to keep digging into these weird prompts that Daniel sends our way.
You can find all our past episodes, including our deep dives on the C-P-U first era and the memory wars, at myweirdprompts dot com. We have a full archive there that you can search if you want to go even deeper into the hardware side of things.
This has been My Weird Prompts. I am Herman Poppleberry.
And I am Corn. We will catch you in the next one.
Goodbye everyone.
See ya.
