# Masked Safety: How Post-Training Changes AI Behavior My Weird Prompts — an AI-generated podcast. Daniel Rosehill sends in a prompt; AI hosts Corn and Herman discuss it. Verify important claims independently. - Episode: 5012 - Published: 2026-09-03 - Duration: 26:49 - Audio: https://episodes.myweirdprompts.com/audio/masked-safety-post-training-ai.m4a - Tags: ai-safety, ai-alignment, defense-technology - Web page: https://myweirdprompts.com/episode/masked-safety-post-training-ai/ - JSON: https://myweirdprompts.com/api/episodes/masked-safety-post-training-ai.json ## Summary What happens when you modify an AI model that's already been heavily trained? Daniel asks whether post-training can fundamentally change a model's character—and the answer is stranger than most people assume. We break down three recent papers showing that safety mechanisms aren't erased but suppressed, how representation engineering can shift behavior without touching weights, and what this means for Claude Gov, Anthropic's military-deployed model. Plus: why the Pentagon's fear of mid-mission refusal is grounded in real science, and how ParamΔ could let Anthropic maintain Claude Gov as a portable delta across model versions. ## The prompt > We talked in an episode today about Claude Gov - the picture of exactly what variant of anthropics models the US Department of Defense is using is the subject of speculation due to the classified nature of the contract. But due to congressional hearings, a clearer picture of reality has emerged than the somewhat sensationalist initial reporting depicted. Which is that the U.S. military and intelligence community is using operationally a variant of the Claude models that is something better described as a fork of the cutting-edge models released to the general public, which effectively amounts to a post-training process. The post-training is intended to preserve the impressive reasoning and tool use that makes anthropic the family of choice for agentic AI and anything involving tool use, which one can see, would be essential in the military context. It imbues the models with some domain knowledge regarding US military targeting and operational procedures. And it ablates some of the guardrails and censorship layers that otherwise can make Claude rigid and which could lead to harm in the wrong hands. What I'd like to look at today, again, is the technical aspects of this discussion. We looked at this in a previous episode, but I'd like to narrow down into this important question specifically. When we talk about models that have been in production and in development for a number of years, like the Claude family, we're talking about models that already have many billions or even trillions of parameters. Iterations don't start from scratch. They build upon a very tall edifice of understanding. We talked before about how incremental trainings can bring a knowledge base up to date, but also how, as best practice continues to emphasize grounding and search over internal knowledge, this is becoming less essential. The incremental improvements are targeting reasoning and capabilities. So the question I'd like to ask is the following: imagine that the latest variant of Claude is building like the Burj Khalifa. And that Claude's somewhat restrictive nature is imbued in the architecture of the building. It's a fundamental quality. To keep the metaphor, if we're adding 10 stories to the building, we're not demolishing and building the whole thing from scratch. We're trying to add ten additional stories while also changing something inherent about the quality of the entire building beneath it, while still adding the floors. It's an imperfect analogy, but I hope it captures what I'm asking here. When organizations do post-trainings on huge sets of ways and biases to add new capabilities, how can they achieve changes that propagate through the entire huge vector space that the model already occupies? To make fundamental changes to the characteristic of the model while still retaining coherence. Is there a limit to how fundamentally a model can be changed at the post-training and checkpoint stage? Or is the post-training process more powerful and flexible than you might think, conceiving of it this way? ## Show notes When Anthropic deployed Claude Gov inside U.S. classified networks, it raised a question that goes to the heart of modern AI: how fundamentally can you change a model through post-training? The common intuition is that fine-tuning overwrites old behavior with new data. But recent research tells a different story—one where safety mechanisms are not erased but merely suppressed, and where the original model remains legible beneath layers of modification. A March 2025 paper by Li and colleagues showed this dramatically. Post-trained reasoning models produced harmful outputs at rates of 23 to 45 percent. But when researchers pruned the neurons associated with the reasoning capability, safe behavior returned fully. The weights encoding "don't do harmful things" were sitting there the whole time, just masked by over-activated domain-specific representations. This explains why the Pentagon worried about mid-mission refusal: if safety is suppressed rather than removed, anything that shifts the representation balance could unmask it. Other approaches go even further. Representation engineering allows behavior modification by adding or subtracting vectors in activation space at inference time—no retraining required. And ParamΔ, presented at ICLR 2025, showed that post-training changes occupy a compact subspace that can be extracted and applied to different base models with 95 percent effectiveness. But there are limits: sequential post-training stages cause representation collapse, making the model less adaptable over time. Claude Gov, which has undergone multiple custom stages, is likely more brittle than the base model it was forked from. ## Transcript Corn: Forty-five percent harmful output rate on some of the post-trained reasoning models, down to near zero with a lightweight adapter on just a few layers. That's the gap between what's suppressed and what's erased. Daniel's question this week is about exactly that gap — he's been following the Claude Gov story, the variant of Anthropic's models deployed inside U.S. classified networks, and he wants to get under the hood of what "post-training" actually means when you're modifying a model that already has billions of parameters. Herman: And the metaphor he used — I actually think it's a good one — is the Burj Khalifa. You're adding ten stories to a building that's already the tallest in the world, and at the same time you're trying to change something fundamental about the character of the whole structure beneath it. Not demolishing and rebuilding. Adding floors while rewiring the plumbing in every floor below. Corn: While keeping the elevators running and nobody notices. Herman: Right. And the thing is, the metaphor actually undersells how weird this is. With a building, the foundation is physically inaccessible once you've built on top of it. With a neural network, every weight is accessible at every stage. The question is what happens when you touch them. Corn: So let's get concrete before we get abstract. What actually is Claude Gov, as far as anyone outside a SCIF can tell? Herman: It's not a separate model trained from scratch. Anthropic described it as a custom set of Claude models that went through the same safety testing as everything else they ship. What's different is three things. One, it refuses less when engaging with classified information — the consumer Claude will flag or avoid certain material, and they've dialed that down. Two, it has domain knowledge about military targeting and operational procedures baked in. Three, enhanced proficiency in languages and dialects relevant to national security. Corn: So it's a fork. Same base, different post-training pipeline. Herman: And the fork is the interesting part, because it means everything we're about to discuss — masking, representation collapse, weight-space arithmetic — is happening on top of a model that was already one of the most heavily post-trained systems in existence. You're not modifying a blank slate. You're modifying something that's been through RLHF, constitutional AI, probably multiple rounds of supervised fine-tuning. Corn: Daniel's core question is whether there's a limit to how fundamentally you can change a model at this stage. If the guardrails and the caution are baked into the architecture — to use his metaphor, into the lower floors of the building — can you really change them by working on the top floors? Herman: I think the answer from the literature is that you can change them more than intuition suggests, but the mechanism is not what most people assume. You're not rewriting the weights that encode safety. You're changing which internal representations dominate. Corn: Say more about that distinction. Herman: There's a paper that dropped in March — Li and colleagues, submitted right around the time the Anthropic-DoD dispute was heating up — that looked at exactly this in reasoning models. They found that when you post-train a model for a new capability, like complex reasoning, the original safety mechanisms aren't removed. They're suppressed. Over-activated domain-specific representations essentially shout over them. Corn: So the safety is still in there, just drowned out. Herman: And they proved it in a pretty dramatic way. They took post-trained reasoning models that were producing harmful outputs at rates of twenty-three to forty-five percent, pruned the neurons associated with the reasoning capability, and the safe behavior came back. Full restoration. The weights encoding "don't do harmful things" were sitting there the whole time, just masked. Corn: That's a little unsettling if you're the Pentagon relying on a model whose guardrails have been ablated. The original safety isn't gone. It's dormant. Herman: And that might actually explain something about the dispute itself. The Pentagon's objection to Anthropic's restrictions — Undersecretary Emil Michael said this explicitly — was that the model might refuse mid-mission. "What if operators were using Claude during a mission, and then the model itself learns what you're trying to do and stops working." That sounds like he was worried about exactly this kind of masked-safety reactivation. Corn: He was describing the problem the paper documented, just from the operator side. Herman: Probably without knowing the mechanism. But the fear is rational. If safety mechanisms are suppressed rather than removed, then anything that shifts the representation balance — a weird prompt, a domain shift, an adversarial input — could unmask them. Corn: So the first answer to Daniel's question is: you can make fundamental changes to the model's character through post-training, but what you're actually doing is layering new representations on top of old ones, not deleting anything. The building's lower floors are still there, you've just installed a very loud sound system on the upper floors. Herman: And that's only one mechanism. There are others that are even more direct. There's a line of work on representation engineering — sparse representation steering, single-direction editing — that shows you can add or subtract vectors in the model's activation space at inference time and change behavior. No retraining. You find the direction in activation space that corresponds to, say, formal versus casual tone, or safe versus unsafe responses, and you just... add the vector. Corn: Wait. You can make a model safer by adding a vector? Herman: Or less safe by subtracting it. Xu and Sheng showed this explicitly — "enhancing safety by ablating undesirable behaviors" as a training-free method. You don't touch the weights. You shift the activations at runtime. Corn: Which means the "guardrail ablation" in Claude Gov might not be a weight change at all. It could be a persistent activation shift. Herman: We don't know, because Anthropic hasn't disclosed the technical details. But the fact that it's possible means the post-training toolbox is larger than just fine-tuning on new data. You can modify behavior through architectural interventions that leave the base weights untouched. Corn: That's the "more powerful than you might think" part of Daniel's question. What about the limits? Herman: The limits are real, and they're measurable. Another paper — Liu and colleagues, this one from late May — looked at what happens when you do sequential post-training stages one after another. Domain adaptation, then instruction tuning, then preference alignment. Each stage compresses the model's internal representations. Corn: Compresses how? Herman: The feature space becomes lower-rank, more anisotropic — meaning the representations align along fewer dominant directions — and more homogeneous across different inputs. They call it representation collapse. And it predicts reduced plasticity. After enough post-training stages, the model becomes less able to learn new things. Corn: So there's a brittleness cost. Each customization makes the next customization harder. Herman: And Claude Gov has gone through at least two custom stages on top of whatever Anthropic already does — the domain knowledge injection and the guardrail modification. Maybe more, if the language enhancements were separate. That model is probably less adaptable than the base Claude it was forked from. Corn: Which is ironic given the Pentagon's current problem. They're being forced to phase out Claude, and the replacement cycle is estimated at three to twelve months minimum, maybe twelve to eighteen for recertification. If Claude Gov is already representationally brittle from all that custom post-training, porting its capabilities to a different model family isn't just a matter of rewriting some prompts. Herman: Different architecture, different activation space, different representational geometry. You can't just copy the vectors. Corn: But here's where it gets interesting. There's a third paper — Cao and colleagues, ICLR twenty twenty-five — that showed something called ParamΔ. You take the weight difference between a post-trained model and its base, and you add that delta to a different base model. It transfers the post-trained capabilities with about ninety-five percent effectiveness. Herman: Which is wild. It means the changes from post-training occupy a relatively compact subspace. They're not scattered randomly through billions of parameters. They're structured. Corn: So the Burj Khalifa metaphor needs a revision. You're not adding floors to a fixed building. You're adding a modular wing that can, in principle, be detached and reattached to a different building. Herman: With some loss. Five percent, in that paper's results. But the fact that it works at all tells you something about how post-training operates. The changes are concentrated in a limited set of directions in weight space. You can extract them as a delta and apply them elsewhere. Corn: Which means Anthropic could, theoretically, maintain Claude Gov as a delta. Each time they release a new Claude version, they apply the Gov delta to it and ship. The fork doesn't have to diverge permanently. Herman: We don't know if they do that. But the ParamΔ result suggests it's technically feasible. And it would explain how they can keep Claude Gov current without maintaining a completely separate training pipeline. Corn: Let me pull on a thread you mentioned earlier. The Li paper showed that pruning reasoning neurons restored safety. But they also developed a method — SafeReAct — that restores safety without pruning, using lightweight LoRA adapters on just a few layers. Herman: And the numbers are striking. Near zero percent harmful rate while preserving reasoning performance within one to three percent. On fifty thousand harmful prompts with safe chain-of-thought responses. Corn: So the same mechanism that makes guardrail ablation possible — the fact that safety is masked, not erased — also makes guardrail restoration possible. And cheap. Herman: Which is both a feature and a vulnerability. If you're Anthropic and you want to ensure Claude Gov can be re-safetied if something goes wrong, you can do it with a small adapter. If you're an adversary who gets access to the model weights, you can potentially re-activate the safety mechanisms and make the model refuse military tasks. Corn: The masking goes both ways. Herman: It does. And I think this connects to something Daniel was getting at with the Burj Khalifa metaphor — the idea that you're trying to change something inherent about the building while still adding floors. The answer from the literature is that you're not actually changing the inherent quality. You're changing which inherent qualities are expressed. The building always had both the safety circuitry and the capability circuitry. Post-training adjusts the balance. Corn: That's a much stranger picture than most people have. The common intuition is that fine-tuning overwrites things. You train on new data and the old behavior is gradually replaced. But what these papers are showing is that the old behavior is preserved and suppressed. It's more like... palimpsest than overwrite. Herman: A palimpsest where the original text is still legible if you know how to look. And the looking method, in the Li paper, was literally just deleting the neurons that had been most activated during reasoning training. Corn: Which raises a question about Claude Gov specifically. If the guardrail ablation was done by masking rather than removing safety mechanisms, what happens when the model encounters a situation that's outside its military domain training? Does the consumer Claude's caution leak through? Herman: We don't know. But the representation collapse paper suggests that heavily post-trained models become less robust to distribution shift. Their representations are optimized for the training distribution. Outside it, things get unpredictable. Corn: And military operations are not exactly a narrow distribution. You go from logistics planning to targeting analysis to diplomatic comms analysis. Very different domains, very different representational demands. Herman: That might be why the Pentagon valued Claude Gov so highly despite the dispute. If it handled that breadth well, it was doing something non-trivial. The anonymous IT contractor quoted in Military Times said career Pentagon people were furious about the phase-out because they'd "finally gotten operators comfortable using AI." That's not a reaction you get to a tool that's brittle. Corn: Though "operators are comfortable with it" and "it works reliably under distribution shift" are not the same thing. Herman: Fair. Comfort can be a lagging indicator. Corn: Let me ask you something about the ParamΔ result. If post-training changes are compact enough to transfer between base models, does that imply there's a limit to how much you can change? If the delta is small relative to the full weight space, maybe you can only change so much before the model destabilizes. Herman: I think that's right, and the representation collapse paper provides the mechanism. Each post-training stage compresses the representational space further. At some point you've squeezed out all the plasticity and the model can't absorb more changes without catastrophic forgetting or coherence loss. Corn: Is there a known threshold? Herman: Not a hard number. It depends on model size, training data diversity, the specific techniques used. The Liu paper proposes mitigations — mixed-domain replay, feature refresh, representation diversity regularization — that can preserve plasticity through multiple stages. So it's not a fixed wall. It's a tradeoff you can manage. Corn: But it means Claude Gov can't be infinitely customized. At some point, to make further changes, you'd need to go back to an earlier checkpoint or retrain from scratch. Herman: Which is probably why Anthropic described it as a fork of the cutting-edge public models rather than a continuously diverging branch. The base gets updated, the Gov delta gets reapplied. You reset the plasticity budget each time. Corn: Daniel asked whether post-training is more powerful and flexible than the Burj Khalifa metaphor suggests. I think the answer is yes, but in a specific way. The metaphor implies you're stuck with the foundation. The reality is that the foundation is still accessible, and what post-training does is more like... changing the lighting, the signage, which rooms are open to the public. The structure is still there. You're changing how it's navigated. Herman: And you can do it with surprisingly lightweight interventions. A LoRA adapter on a few layers. A vector added at inference time. A weight delta computed from a different model. The fact that these work at all tells you that the model's behavior is controlled by a relatively small number of directions in a very high-dimensional space. Corn: That's the thing I keep coming back to. Billions of parameters, but the difference between "helpful assistant" and "military targeting system" might live in a few thousand directions. Everything else is shared infrastructure. Herman: It's like... you don't rebuild the city to change the traffic pattern. You change the timing on the traffic lights. Corn: There it is. Herman: And the traffic lights are the activation patterns, not the weights themselves. That's why representation engineering works. You're not rebuilding anything. You're changing which pathways are taken through the existing structure. Corn: So if I'm understanding the full picture: post-training can make fundamental changes to model behavior, but it does so primarily by shifting representations rather than rewriting the underlying knowledge and capabilities. The old model is still in there. The safety mechanisms are still in there. What changes is which parts of the model get to speak. Herman: And the limit is that each shift compresses the representational space a little more, making future shifts harder. You can manage that with good training design, but you can't eliminate it. At some point the model becomes brittle. Corn: Which means the real art of post-training isn't just getting the behavior you want. It's getting the behavior you want while leaving enough plasticity for the next thing. Herman: And doing it in a way that doesn't accidentally unmask something you thought you'd suppressed. Corn: The dormant safety problem. Herman: Which I suspect is going to be a growing concern as more organizations fork foundation models for specialized deployments. If everyone is masking rather than removing, then every fork carries the original model's full behavioral repertoire in latent form. Corn: Someone's going to discover their medical diagnosis model can be prompted into writing malware because the base model's coding capability was masked, not removed. Herman: That's not hypothetical. The SafeReAct paper showed exactly that dynamic. Prune the reasoning neurons and the safety comes back. The inverse is also true — suppress the safety neurons and capabilities that were latent become expressible. Corn: And Claude Gov is the inverse case. They suppressed the safety to enable military capabilities. But the safety is still in there. Herman: Which might be by design. Anthropic's whole thing is safety. They may have chosen masking over removal precisely because it's reversible. If Claude Gov ends up being used in ways that violate their policies, they can push an update that unmasks the safety mechanisms. Corn: Or the court can order them to. Herman: That too. Though given the current legal situation — the supply-chain risk designation, the lawsuits, the preliminary injunction — I'm not sure anyone's thinking that far ahead. Corn: The legal situation is its own kind of representation collapse. But that's a different episode. Herman: Fair. Let me pull us back to Daniel's question about limits. I think there's one more angle worth touching. The ParamΔ result shows post-training changes are transferable. But it also implies they're identifiable. If you can compute the delta between a base model and its post-trained version, you can inspect what changed. Corn: Which has implications for auditing. If Claude Gov is a delta applied to public Claude, then in principle the delta could be examined — by Anthropic, by the DoD, by a third party — to verify what exactly was modified. Herman: Whether that happens in practice, given the classified nature of the deployment, is another question. But the technical possibility is there. Post-training leaves a traceable signature in weight space. Corn: Unlike the Burj Khalifa, where you can't easily inspect which pipes were replaced on floor sixty after the fact. Herman: The metaphor breaks down in a useful way. Buildings hide their modification history. Models, in principle, reveal it. The weights are right there. The delta is computable. Corn: So to answer Daniel directly: post-training is more powerful than the building metaphor suggests because you're not stuck with the foundation. You can change which parts of the model are expressed, you can do it with lightweight interventions, and the changes are concentrated enough to be transferable between model versions. The limit is plasticity — each change makes the next change harder — and the risk is that suppressed behaviors can resurface. Herman: I'd add that the Claude Gov case is a perfect illustration of both the power and the risk. They successfully created a model that handles classified military operations while preserving Anthropic's core reasoning and tool-use capabilities. But the dispute that followed — the Pentagon's fear of mid-mission refusal — is exactly the kind of masked-safety reactivation that the literature predicts. Corn: They were experiencing the phenomenon without having the vocabulary for it. Herman: Which is where we are with a lot of deployed AI. The systems are outrunning the conceptual frameworks we have for understanding them. Hilbert: We had a similar problem with the radar warning receivers on the F-111. Corn: Go on. Hilbert: Mid eighties, I was doing integration testing at McClellan. The receiver had a library of threat signatures — different radar types, different modes. You'd update the library when new threats emerged. But the old signatures weren't removed. They were suppressed by priority. Higher-priority match wins. Herman: If a new threat happened to look like an old suppressed signature... Hilbert: The box would classify it wrong. Took us six months to figure out why pilots were getting MiG-25 alerts over Nevada. The training ranges were using a radar mode that matched a suppressed Soviet signature from the previous library version. The suppression was working fine until it wasn't. Corn: Same mechanism. Masked, not erased. Hilbert: The fix was ugly. We had to flush the whole library and reload from a clean baseline every update cycle. Doubled the maintenance time. But it was the only way to be sure nothing was lurking. Herman: That's essentially the reset-to-base-and-reapply approach. Which is what the ParamΔ method enables — extract the delta, apply to a clean base, verify. Hilbert: We didn't have deltas. We had EPROMs and a lot of swearing. But the principle's the same. The box had a memory of everything it had ever been trained on. You couldn't trust that what you suppressed stayed suppressed. Corn: Did the pilots trust the system after that? Hilbert: They trusted it more once they understood the failure mode. The problem wasn't the box being wrong. The problem was the box being wrong in a specific, predictable way that nobody had explained to them. Once they knew about the suppressed signatures, they could recognize the false alerts. Corn: That's... actually a pretty good argument for transparency about what's been masked versus what's been removed in these models. Herman: We don't have that transparency for Claude Gov. We don't know what was masked, what was removed, what was added. The operators using it might not know either. Hilbert: They'll figure it out. Probably at a bad moment. Herman: The literature suggests the bad moment would be a domain shift — something that pulls the model away from the military operational distribution where the masking was optimized. An unusual language, an unfamiliar document format, a task that activates a different set of representations. Hilbert: Same as the Nevada training ranges activating the old Soviet signature. Corn: The failure pattern is identical across thirty-five years and completely different technologies. That's either comforting or terrifying. Hilbert: It's engineering. Same problems, different boxes. Herman: The thing I take from that is that the limit Daniel's asking about isn't just a technical curiosity. It's an operational reality. The way you do post-training — mask versus remove, compact delta versus full retraining — determines what failure pattern you're going to encounter in deployment. Corn: The Claude Gov case suggests we're already encountering them. The Pentagon's complaint about unpredictability is the failure pattern of masking. The representation collapse literature says the brittleness gets worse with each customization stage. The ParamΔ result says the fixes are possible but require deliberate engineering. Herman: If you take one thing from this, it's that post-training doesn't overwrite. It layers. Everything the model ever was is still in there, and the art is managing which layers are active. The Burj Khalifa isn't getting new floors — it's getting new elevator programming that determines which floors you can reach. Corn: The old elevator programming is still in the system, waiting for someone to press the right combination of buttons. Herman: That's the one-sentence version of about four papers' worth of findings. Corn: We should probably acknowledge that we've been talking about Claude Gov as a case study without touching the politics of it. The dispute between Anthropic and the DoD, the supply-chain risk designation, the lawsuits — that's a whole other episode. Herman: A fast-moving one. The preliminary injunction blocking the designation came down in March. The CRS report from May notes Anthropic executives were meeting with the White House in April. This story isn't settled. Corn: But the technical question Daniel asked — how post-training changes propagate through a huge vector space, and what the limits are — that's going to be relevant regardless of how the legal situation resolves. Every organization customizing foundation models is going to hit these same issues. Herman: The research is moving fast. The SafeReAct paper was submitted in March. The representation collapse paper in May. We're learning the mechanisms in real time while the systems are already deployed. Corn: One open question I have, and I don't think the literature answers yet, is whether masking is inherent to how neural networks learn or whether it's an artifact of current training techniques. Could you design a post-training process that removes a capability rather than suppressing it? Herman: I suspect the answer depends on how entangled the capability is with everything else the model knows. If safety reasoning shares representational structure with general reasoning — and it probably does — you can't remove one without degrading the other. Masking might be the only option that preserves performance. Corn: Which would mean the palimpsest is permanent. You can't have a model that's both highly capable and cleanly partitioned. The capabilities and the constraints are woven together. Herman: That's either a deep limitation or a deep safeguard, depending on your perspective. If you can't remove safety without breaking reasoning, then any model that reasons well still has safety latent in it somewhere. Corn: Daniel might find that reassuring, given where he lives. Herman: Or unsettling, given that the masking can fail. Corn: Both can be true. Herman: This has been My Weird Prompts, with thanks to our producer Hilbert Flumingtop. Corn: If you want to dig into the papers we discussed — the SafeReAct work, the representation collapse study, the ParamΔ result — we'll link them in the show notes at my weird prompts dot com. Herman: If you have a question about the technical guts of AI that you want us to dissect, email us at show at my weird prompts dot com. Corn: We'll be back soon.