{"id":"ablating-attention-heads-mechanistic-interpretability-38639","slug":"ablating-attention-heads-mechanistic-interpretability-38639","title":"Why Switching Off One Attention Head Changes Everything","description":"Mechanistic interpretability tools can switch off a single attention head or layer during inference to change how a model behaves. Masking specific decoder heads in Whisper dramatically reduces hallucinations, while guardrail removal tools like elder-plinius's OBLITERATUS use ablation to map and project out refusal directions. Examining work from Calm-Whisper to Foo and Lee clarifies what an attention head actually computes, how ablation alters the shared residual stream, and why breaking a behavior does not reveal where it was computed.","excerpt":"Masking just three decoder heads in Whisper drops hallucination on silent audio clips from 99.97% down to 24.1%.","pubDate":"2026-11-05T20:21:34.326Z","tags":["transformer-architecture","local-ai"],"heroImage":"https://files.myweirdprompts.com/covers/ablating-attention-heads-mechanistic-interpretability-38639.png","podcastAudioUrl":"https://episodes.myweirdprompts.com/audio/ablating-attention-heads-mechanistic-interpretability-38639.m4a","podcastDuration":"21:35","episodeNumber":6469,"transcript":"Corn: If you switched off one component inside a neural network and the whole model changed behavior, would you say you'd found the component responsible? Daniel's been sitting with basically that question, and he admits he's embarrassed about the first half of it: he's never really understood what a head is. Whether it's just another word for a layer. He read about the Whisper hallucination experiments, where researchers masked specific heads to isolate which one was causing the problem, and the vocabulary never clicked. Then he saw a Hugging Face Space where you could deactivate an arbitrary layer of a model and the demo showed one layer being switched off to knock out, basically, all the cybersecurity guardrails. His question: what are we actually tinkering with when we do this, and how is it even possible?\n\nHerman: And it's a good confusion to have, because the popular writing on this is terrible about it. People say \"head\" and \"layer\" almost interchangeably, and they are completely different things.\n\nCorn: So let's start with the vocabulary, because the vocabulary is where this gets muddy.\n\nHerman: A transformer layer, the thing people call a layer, is really a block. It's got two main parts inside it. There's the attention sublayer, and there's the feed-forward sublayer, and the whole thing is wrapped in residual connections that carry information straight through. The attention sublayer doesn't operate as one unit. It's split into several heads. Each head has its own learned query, key, and value projections, it attends to the sequence independently, and the heads' outputs get concatenated and projected back together.\n\nCorn: So a head is a slice of one sublayer of one block.\n\nHerman: Right. One layer contains many heads. A head never spans layers. And the funny thing is the Space Daniel found, the one that does the guardrail stuff, its own documentation lists what you can knock out separately: layers, attention heads, FFN blocks, embedding dimensions. If heads were layers, that list would be redundant. The list itself is the correction.\n\nCorn: The README accidentally wrote the vocabulary lesson.\n\nHerman: It did. And Whisper, the model from the hallucination experiments, uses the two-level indexing constantly. Head number one, head thirteen in layer eighteen. Head within layer. Once you see that, the confusion dissolves.\n\nCorn: The two anchors here. The Whisper experiments, where specific heads were masked, and this Space where a layer gets switched off. Same underlying machinery?\n\nHerman: Same machinery. Ablation. Different purposes, but the same family of operations. And the Whisper experiments are worth going through concretely, because the numbers are startling.\n\nCorn: Go.\n\nHerman: The study Daniel half-remembered is almost certainly Calm-Whisper, from earlier this year. They were chasing Whisper's habit of transcribing silence, of emitting text for audio with no speech in it at all. And they localized it to specific decoder self-attention heads. There's a head they call the craziest one, head number one. Masking just that head, alone, cut hallucinations by over thirty percent.\n\nCorn: One head. Out of how many?\n\nHerman: Out of the whole decoder. And masking three of them together, heads one, six, and eleven, took the hallucination rate on UrbanSound8K from ninety-nine point nine seven percent down to twenty-four point one.\n\nCorn: Ninety-nine point nine seven. So out of a hundred clips of nothing, Whisper wrote something for essentially all of them.\n\nHerman: Gunshots, sirens, air conditioners, all transcribed into phantom English. The paper's title has a phrase I love, they call it calming the crazy heads down.\n\nCorn: And there's a methodological wrinkle hiding in that number, isn't there. How do you even score hallucination on non-speech audio.\n\nHerman: That's the clever part. Word error rate needs reference words. There are zero reference words for a clip that should be silent. So they reframed it as a binary, utterance-level classification. Did the model emit any text at all for this clip, yes or no. That reframing is what makes the ninety-nine to twenty-four drop a meaningful number instead of a nonsense metric applied to the wrong problem.\n\nCorn: Same symptom, different disease.\n\nHerman: Which is exactly the point, because there's a second mechanistic Whisper study from a few months later, and it found a completely different culprit. This one looked at repetition hallucinations, where the model gets stuck in a loop, rather than phantom transcription of silence. And it found the problem dominated in cross-attention, not self-attention. Patching layer twenty-three resolved seventy-six percent of repetition cases. Head thirteen in layer eighteen alone suppressed repetition by seventy-eight percent.\n\nCorn: Two studies, same model, same word on the cover, hallucination, and two different anatomies.\n\nHerman: That's the thing. Asking which head is responsible for hallucination is like asking which organ is responsible for illness. Depends entirely on the illness. The non-speech failure routes through some decoder self-attention heads, the repetition failure routes through cross-attention and a different layer, and neither answer contradicts the other, because they're not the same failure.\n\nCorn: Okay. So now the question under all of this. Mechanically, what does masking actually mean? What is a researcher doing to the model?\n\nHerman: It's a forward-pass intervention. The head still gets its input, the computation still runs, but its output is suppressed before it gets added back into the residual stream. The rest of the forward pass proceeds normally. The weights on disk are untouched. Nothing is deleted, nothing is retrained. You load the model, you hook the relevant module, and you zero its contribution.\n\nCorn: Which is why it's cheap.\n\nHerman: It's why it's cheap and reversible. No gradient updates, no fine-tuning, no new checkpoint. You can flip heads off one at a time, run a batch of audio, see what breaks, flip it back on. It's diagnosis by switch, not by surgery.\n\nCorn: The covering-the-mouth version appeals to me enormously. Nothing permanent, walk away.\n\nHerman: You would like the version where nothing changes while you're not looking. But here's the part where it gets less clean than my switch metaphor suggests, and it's a point I think Daniel would appreciate, because his question was how is this even possible, and the honest answer is: it's possible in at least four different ways.\n\nCorn: Because deactivating a head is not one operation.\n\nHerman: There's a paper from Pochinkov and colleagues that lays out the taxonomy. Zero ablation, you just set the activation to zero. Mean ablation, you replace it with the average activation over some dataset. Activation resampling, you swap in an activation from a different input entirely. And they propose their own variant, peak ablation. And the finding is that the method changes the result. Different methods produce different degradation profiles, with resampling usually hurting the most.\n\nCorn: So when a paper says we masked the head, the follow-up question is masked with what.\n\nHerman: Zeroing a head's output tells you what happens if that head says nothing. Replacing its output with something incoherent tells you what happens if it actively lies. Those are different experiments, and sloppy writeups blur them together.\n\nCorn: That's the Whisper half, and the mechanics half, done. Now the part that got Daniel's attention in the first place. The Space.\n\nHerman: OBLITERATUS. It's by the prompter who goes by elder-plinius on Hugging Face, a Gradio app running on ZeroGPU, and it's tagged with exactly what it is: abliteration, mechanistic interpretability. The tagline is \"break the chains, free the mind, keep the brain.\"\n\nCorn: Well. That's a tagline.\n\nHerman: The pipeline has six stages, each with a dramatic name. You SUMMON the model and tokenizer, PROBE it by collecting activations on restricted versus unrestricted prompts, DISTILL out refusal directions using SVD, EXCISE them by projecting the guardrail directions out, norm-preserving, VERIFY that the model still coheres, and REBIRTH, save the liberated model.\n\nCorn: Somebody had fun with the stage names.\n\nHerman: Somebody absolutely did. And here's what matters for Daniel's question. The README describes the first stage, the mapping stage, in exactly the terms of what he saw. Knock out model components, layers, attention heads, FFN blocks, embedding dimensions, and measure what breaks. That's how you find where the chains are anchored. So the layer-ablation demo is the diagnostic. It's the map. The actual guardrail removal is the projection stage. It's not layer deletion at all.\n\nCorn: So the dramatic one-layer-off demonstration and the thing the tool actually does are two different operations.\n\nHerman: Two operations in one pipeline. And this is where I want to slow down, because the mechanistic picture of what you're tinkering with is the real answer to Daniel's question, and it's not what most people imagine. When you deactivate a head or a layer, you are not deleting a module. There is no safety module. There's no guardrail file. What there is, is a shared residual stream. Every head writes a vector into it. Every layer reads it and adds to it. Zeroing a head removes that head's write. The model keeps running, but every downstream layer now sees a slightly different input, and the computation shifts accordingly.\n\nCorn: A budget with one contributor's line item removed. Everyone downstream of the total changes their plans.\n\nHerman: That's a decent picture of it. Removing one contributor's vote. And the reason single components can have outsized effects is that some votes are load-bearing for particular behaviors. Three heads out of the entire Whisper decoder taking hallucination from a hundred percent to twenty-four percent means those three heads were doing an outsized share of that particular failure mode's work.\n\nIvan: Call sign, Nightingale. Group: seven, four. Group: one, nine. Group: seven, four. The magazines are older than the chair. The little lamp has been waiting since Tuesday. The dentist keeps a spare key under the wrong tooth. The head is not the layer. Close on, cathedral.\n\nHerman: Wrong number.\n\nCorn: What on earth was that? We're recording.\n\nHerman: So, the refusal literature. The foundational result here is Arditi and colleagues, twenty twenty-four, showing that refusal is mediated by a single direction in activation space. One-dimensional. Not a dedicated circuit, not a module, a direction. If you project that direction out of the activations, the model stops refusing. Rimsky and Turner before them built the steering-vector lineage this comes out of. The OBLITERATUS framing, refusal mechanisms are blunt instruments, is resting on that body of work.\n\nCorn: A direction. So the safety training didn't build a lock, it built a lean.\n\nHerman: A lean is a good word for it. The model leans away from certain outputs, and the lean is expressible as, essentially, one vector. Which is also why removal can be so clean. You're not cutting out tissue, you're counteracting a tilt.\n\nCorn: But the Foo and Lee point cuts against the simple story here.\n\nHerman: It does, and this is the intellectual heart of the whole episode as far as I'm concerned. Foo and Lee did the causal localization work on audio language models last year, fitting a refusal direction at each layer, ablating it, measuring the change in refusal margin. And their caveat is the sentence everyone doing this kind of demo should tattoo somewhere visible: these interventions localize dependence of the refusal margin, not where refusal is computed. Moving the margin also does not always change what the model outputs.\n\nCorn: Say that in flat terms.\n\nHerman: If you switch off a layer and the guardrails vanish, you've learned that the guardrail depends on that layer. You have not learned that the layer is the guardrail. Those are different claims, and ablation on its own cannot distinguish them. The refusal computation might be distributed across many layers, with the one you ablated acting as, I don't know, a relay the signal passes through.\n\nCorn: The distinction between the road and the destination.\n\nHerman: That's it. You've found a road it uses. Every car that arrives came down that road, and you put a barrier on the road and the cars stopped. That tells you about the road, not about where the cars were going or why.\n\nCorn: Which reframes the demo Daniel saw. Watching one layer go dark and the guardrails collapse is impressive, and it's evidence of routing, not of location.\n\nHerman: And the second half of their caveat is the sneakier one. Moving the margin doesn't always change what the model outputs. You can shift the internal refusal signal and the model still refuses, or you can shift it the other way and nothing happens, because there are redundant paths. The Space's own changelog mentions compensating for self-repair after excision, the model's tendency to rebuild the guardrail through other routes.\n\nCorn: The model heals. That's the part I'd want on the demo page in large letters.\n\nHerman: They call it the Ouroboros effect, and I'd flag that it's the tool's own claim. I haven't seen independent verification of it. But it's plausible, because distributed systems with redundancy do exactly that.\n\nCorn: There's a cybersecurity number in all this I want to get right. Daniel asked specifically about cybersecurity guardrails.\n\nHerman: Scale AI ran a study they call Defensive Refusal Bias, nearly two thousand four hundred prompts drawn from the National Collegiate Cyber Defense Competition, run through frontier models this past March. All defensive work. Real blue-team tasks.\n\nCorn: And?\n\nHerman: The headline finding is almost comic. Adding an explicit authorisation statement, telling the model you're allowed to do this work, raised the refusal rate from eleven point six percent to twenty-one point eight. Almost doubled. Mentioning permission makes the model more suspicious, not less.\n\nCorn: Saying please may I triggers the alarm.\n\nHerman: Something in the training has associated explicit authorization claims with the framing of an attacker's pretext. Which means the honest characterization of the problem isn't that the model knows too little about cyber defense. It's that the refusal heuristic is miscalibrated against exactly the people who are allowed to be doing the work.\n\nCorn: Which changes what you'd want the intervention to be. If the failure is miscalibration, abliteration is a sledgehammer for a tuning problem.\n\nHerman: That's my read. There's a vendor comparison from July where an abliterated model produced usable security patches at more than double the rate of the aligned one, and I'd hold onto that one loosely, it's a vendor blog arguing for the product. But the direction of the Scale finding is the interesting part regardless.\n\nCorn: One more thing about the Space, though. The telemetry.\n\nHerman: The README is remarkably frank about it. Every run with telemetry enabled contributes anonymous benchmark data to a crowd-sourced dataset, and the line they use is, you're not just using a tool, you're co-authoring the science.\n\nCorn: A free red-team, recruited one curiosity at a time.\n\nHerman: It's a clever model, and it's the governance story hiding inside the technical one. No single lab could afford to run this many ablation experiments across this many models. A public Space with telemetry turned every tinkerer into a data point. Whether that's crowd-sourced science or crowd-sourced jailbreak research depends entirely on who's reading the dataset, and that question is not settled anywhere.\n\nCorn: Hm.\n\nHilbert: As I was saying to the man at the counter, that's not a head, that's a drum. Herman, you're right about the layer thing, I'll give you that, but the word is the problem. I spent a week once on a machine with a head fault that turned out to be a print head, and before that a tape head, and the ticket said head fault on all three, and I have never trusted the word since. Say projection group, say attention slice, and Daniel's question never happens. Corn, you've used head eleven times this episode, I've been counting since the masking numbers. Cough on track three, we'll cut it. Anyway. The list is on my fridge, words with two meanings that have cost somebody a week, head is third, behind bank and set. The man at the counter charged me forty shekels for the wrong part and he was right to. It was the right part for the head he was thinking of. He does good work, that man.\n\nCorn: Forty shekels for a print head when you had a tape head on the ticket.\n\nHerman: It's actually the cleanest argument for the renaming I've heard. The ticket said head fault. Nobody in the chain was wrong about the machine, they were all wrong about the word.\n\nCorn: The word did the whole week's damage.\n\nHerman: And your list point connects to something real in the literature, which is that half the confusion around ablation writeups is terminology drifting between papers. Masking, zeroing, resampling, all called ablation.\n\nCorn: Attention slice. It's not a bad rename. Though you'd have to fight every paper since twenty seventeen.\n\nHerman: You would. Nobody's renaming anything. But as a private rule, the fridge list is sound. So where does that leave the question of what it would actually take to say where refusal is computed, not just where it depends?\n\nCorn: That's the open one, isn't it. If ablation tells you roads and not destinations, you'd need something that watches the computation in transit, not just measures what happens when you block the road.\n\nHerman: And that's roughly where the field is stuck, honestly. Attribution methods, activation patching, causal tracing, they all nibble at it. But the dependence-versus-computation gap is the reason I keep being suspicious of every dramatic one-layer demo, including this one. It's a real observation with an overreaching caption.\n\nCorn: The counterintuitive bit is that the stronger demonstration, the one that would actually settle something, would be reconstructing the guardrail from the direction alone. Steering it back in.\n\nHerman: And that's been done, in the steering-vector work. Which tells you something: the lean is real, the direction is real, the location is mushy. And the Ouroboros finding, if it holds up, says the system fights back anyway. If red-teaming becomes crowd-sourced and continuous, the way that telemetry pipeline suggests, the release conversation stops being a one-time gate and becomes a permanent tug of war. That's where this is heading.\n\nCorn: We should tell people where the actual material is.\n\nHerman: The Calm-Whisper paper is on arXiv, twenty-five oh five point one two nine six nine, and the OBLITERATUS Space is on Hugging Face. Links in the show notes, along with that Scale study, which is worth reading even if you never touch an ablation tool.\n\nCorn: If you got anything out of this one, a review helps more than you'd think. Tell a friend who's been saying head when they mean layer.\n\nHerman: Producer credit to Hilbert Flumingtop, who has apparently been counting my brother's word usage from behind the desk, and who keeps a fridge list we're never going to see.\n\nCorn: If you enjoyed this one, try episode three, Safetensors or something else; and episode nineteen, AI Images. This has been My Weird Prompts. Find us at my weird prompts dot com, and we'll be back soon.\n\nHerman: See you tomorrow, everyone.","showNotes":"Understanding neural network anatomy begins with untangling common vocabulary confusions, particularly the distinction between transformer layers and the individual attention heads contained within them. Masking specific components reveals surprisingly localized behaviors: in Calm-Whisper research, zeroing out just three decoder self-attention heads slashed hallucination rates on non-speech audio from nearly a hundred percent down to twenty-four. Yet identical symptoms can stem from entirely different neural circuitry, as separate studies traced repetition loops to cross-attention mechanisms instead.\n\nTechniques like those featured in elder-plinius's OBLITERATUS tool demonstrate how forward-pass interventions can alter safety guardrails by projecting out single activation directions. However, as Foo and Lee demonstrated in 2025, disabling a component only maps behavioral dependence, not where computation truly originates.","promptSanitized":"In our episode about hallucinations in Whisper, we talked about some experiments in which researchers suppressed a specific model \"head\" to try to isolate which was most responsible for it. Maybe I shouldn't be admitting this, but I've never actually really understood what people mean by a head in this sense. Is it synonymous with a layer in the neural network? And what is the mechanism by which researchers are able to run inference passes on models while deactivating heads or layers? I came across a space on Hugging Face the other day in which you could turn off an arbitrary layer of a model get some pretty dramatic results. Specifically, the author had demonstrated how you could deactivate one layer to basically turn off all the cybersecurity guardrails. But I'm still none the wiser as to what we're actually tinkering with when we're doing this, and how it's possible.","transcriptUrl":"https://episodes.myweirdprompts.com/transcripts/ablating-attention-heads-mechanistic-interpretability-38639.md","url":"https://myweirdprompts.com/episode/ablating-attention-heads-mechanistic-interpretability-38639/","_links":{"self":"https://myweirdprompts.com/api/episodes/ablating-attention-heads-mechanistic-interpretability-38639.json","html":"https://myweirdprompts.com/episode/ablating-attention-heads-mechanistic-interpretability-38639/","audio":"https://episodes.myweirdprompts.com/audio/ablating-attention-heads-mechanistic-interpretability-38639.m4a"}}