A hundred and two kilobytes. That's the entire customization. The base model is four hundred and three gigabytes of frozen weights, and the thing that makes it a different model is a file smaller than a thumbnail.
And it is not merged. That's the part that keeps surprising people. You load the base, you point at the adapter, and the two run together at inference. The base never changes.
Which is exactly what Daniel wants to talk about. Here's what he wrote in this week. We've done LoRA before, but this time he wants us to zoom out and look at the whole category of model adapters. What is an adapter, actually. Why does Hugging Face treat adapters as a distinct model category. He points us at Tayi Flash Q4 as the worked example and asks us to explain the underlying idea of adapting a foundation model without creating another fully fine-tuned copy. Then he wants the zoo. Beyond LoRA, what other adapter architectures and techniques exist, how do they modify or extend a base model, and what are the real-world use cases. He names bottleneck adapters, prefix tuning, prompt tuning, IA³, and the rest of the parameter-efficient family. And then the question that I think is the actual spine of this episode. Are these all variations on essentially the same idea, or are fundamentally different mechanisms being grouped together under one word. He closes with two more. Why would someone today deliberately choose a non-LoRA adapter. And are adapters best understood as a training technique, a distribution format, or an emerging modular layer in the model ecosystem.
That last one is the one I want to answer most, and I suspect the answer is going to be all three at once.
Let's start with the smallest possible version of the question. What actually gets shipped when someone ships an adapter?
Formally, you take a pretrained model and you split its parameters into two sets. The frozen pretrained weights, call them theta, and a much smaller set of newly introduced parameters, call them phi. Efficient fine-tuning optimizes only phi. Theta never moves. That's the whole idea in one line, and everything else in this episode is a variation on how you choose phi.
AdapterHub's numbers are the cleanest. A typical adapter is around three megabytes per task. A full fine-tuned model is around four hundred and forty megabytes. And the fraction of parameters actually being updated is usually under one percent. Thinking Machines put the intuition better than anyone. "It seems wasteful to use a terabit of weights to represent updates from a gigabit or megabit of training data."
That's a good line.
It's the whole economic argument. You spent a fortune pretraining the base. Then you want it to do one more thing. Do you really need to write down a second copy of every weight to capture what that one thing taught it?
So why does Hugging Face treat adapters as a distinct model category? Because that's the framing Daniel led with, and I want to be careful here, because I think the honest answer is more functional than formal.
You're right to flag that. There's a dedicated Adapters filter on the Hub, and there's a docs page called Using Adapters at Hugging Face. But the reason adapters get their own treatment is that they're add-on artifacts, not standalone weights. You load a base transformers model, then you call load_adapter, then you set active_adapters. The adapter is something you attach. That's the distinction. It's functional. It's not that the Hub has an "adapter" entry sitting next to "text-generation" and "vision" as a formal model type. I looked for that and I don't think it exists in that form.
Good. So the category is real but it's defined by how the artifact is used, not by a taxonomy entry.
Right. And the tooling reflects that. There are two libraries doing the work. The adapters library, which is the successor to adapter-transformers and is fully weight-compatible with it, and the PEFT library. PEFT is the one that says "we only fine-tune a small number of extra parameters on top of a pretrained model, and those extras are lightweight, which makes them convenient to share, store, and load." Transformers wires it in through a mixin on every pretrained model class.
And here's the caveat I want on the table before we go any further, because it's going to pay off later. Hugging Face's own docs admit the word is used loosely. Their words, roughly: the term adapter is usually only applied to bottleneck adapter modules, but most efficient fine-tuning methods follow the same general idea, so in their library the term adapter may refer to any efficient fine-tuning method if not specified otherwise.
They're basically saying "we know this word is doing too much work and we're going with it anyway."
Which sets up Daniel's central question perfectly. So that's the idea. Now here's what it looks like when someone actually ships one.
Tayi Flash Q4. It's a low-bit release, MXFP4, Q4-class, put out by Paulo Lima under Tayi AI. And the structure is the cleanest illustration of the concept I've seen on the Hub. The base is twelve GGUF shards. Four hundred and three point five zero gigabytes decimal, three hundred seventy-five point seven nine gibibytes. That's the frozen part.
And the customization?
One hundred two thousand seven hundred fifty-two bytes. A single GGUF LoRA file, sitting at adapters slash tayi-flash-q4.lora. You load it with the lora-scaled flag at scale one point zero. No merge, no conversion.
Say the parameter numbers, because the ratio is the story.
The base shards contain seven hundred forty-eight billion, four hundred ninety-four million, six hundred sixty-nine thousand, four hundred twenty-four tensor elements. The adapter updates twenty-five thousand, six hundred factor elements. It's rank four, alpha eight, and it targets exactly one module. Block zero, attention query A weight.
One module.
One module. Out of the entire network. And the model card is very deliberate about what that means. Quote: "This is an adapter-trained derivative, not a claim of original backbone pretraining or full-model weight updates."
There's a second line in that card that I think is the single best sentence in this whole episode.
Go ahead.
"The adapter is included but deliberately not merged or requantized: running only the base is not a Tayi run."
That's the distribution-format argument in one sentence. The base and the adapter are two artifacts, and the model is the combination. If you download four hundred gigabytes and skip the hundred-kilobyte file, you don't have the model. You have the substrate.
Which is a strange thing to be able to say about a hundred-kilobyte file. Now, the honesty angle, because this is what makes the card unusual.
They published a regression. The primary metric is MMLU generated-letter exact match. Tayi scored one hundred thirteen out of five hundred eleven, twenty-two point one one percent. The Q4 reference scored one hundred sixteen out of five hundred eleven, twenty-two point seven zero percent. That's a net minus three correct decisions, minus zero point five eight seven percentage points.
And the card's own verdict on that?
"These observations do not support presenting the candidate as a quality-superior or lossless variant."
Most release cards would have found a way to phrase that as a win.
They also published the runtime overhead. Mean request latency seventeen point zero zero five seconds for Tayi versus sixteen point seven zero zero for the reference. Plus one point eight three percent. And then they warn that roughly one point eight percent across four prompts does not establish a universal adapter overhead. They're refusing to let their own number become a marketing claim.
I want to flag the base lineage too, because it's exactly the kind of number people misattribute. The backbone is DeepSeek-V4.1-Flash. Five hundred fifty-two billion backbone parameters, eight billion activated on prefill, sixteen billion on decode, forty-five trillion pretraining tokens per the upstream metadata. And the card lists comparison benchmarks like GPQA Diamond at ninety point nine and Terminal-Bench two point one at ninety point six, but those are upstream reference values. Not Tayi reproductions. The card says so explicitly.
And there's no locally measured Tayi score for any of those. The only local readout is the MMLU number, and it's a regression. That's the honest picture.
So that's the worked example. Now the zoo, because Daniel asked for the zoo and I want every animal described, not just named. Start with the original.
Bottleneck adapters. Houlsby and colleagues, twenty nineteen. This is what "adapter" meant before the word got stretched. You take the hidden state, project it down to a smaller dimension with a down-projection matrix, run it through a nonlinearity, project it back up with an up-projection matrix, and add the result back to the original as a residual. So the output is the up-projection of the nonlinearity of the down-projection of the input, plus the input itself. The key hyperparameter is the bottleneck dimension, set by a reduction factor that's the hidden size divided by the bottleneck size.
Where do they sit in the network?
After attention, after the feed-forward block, or both. That placement is actually how the named variants differ. The Houlsby config puts them after both. The Pfeiffer config puts them after the feed-forward only. There's a parallel variant that runs alongside the layer instead of after it. And a twenty twenty-four variant from Steitz and Roth that refines the design further.
And the language-adapter branch?
That's MAD-X, from Pfeiffer and colleagues in twenty twenty. Language adapters learn language-specific transforms, and there's an invertible adapter sitting after the embedding layer. The invertibility is the trick. It lets you swap one language adapter for another, which gives you zero-shot cross-lingual transfer by literally substituting the module.
Now prefix tuning, because this is where the mechanism changes category.
Prefix tuning is Li and Liang, twenty twenty-one. Instead of adding weights, you prepend trainable prefix vectors to the keys and the values of every attention layer. So you're modifying the activations flowing through attention, not the weight matrices. They reparameterize the prefix through a small bottleneck MLP for training stability, and then you can eject it back to a flat form at inference. Reportedly it matches full fine-tuning with a thousand times fewer parameters, and it does better in low-data settings.
Prompt tuning is the cousin.
Lester and colleagues, same year. Prompt tuning only adds learnable soft prompt tokens at the input embedding layer. Not every layer. Just the input. And its performance scales with model size. The bigger the base, the better the soft prompt works. There's a variant that uses an LSTM prompt encoder and can insert tokens anywhere, and a multitask version.
And here's the thing I want said plainly, because it's the hinge of the whole episode. Neither of these modifies a single weight.
Correct. Zero weight change. Which is why Hugging Face's own tooling splits them out. The mixin that wires PEFT into transformers supports all the non-prompt-learning methods, LoRA, IA3, AdaLoRA and the rest. And the docs say prefix and prompt methods aren't supported because they can't be injected into a torch module. That's the tell. If your method doesn't produce a module you can graft onto the network, it isn't the same kind of thing.
IA³ next. Daniel specifically called it out.
IA³, from Liu and colleagues in twenty twenty-two, out of the T-Few work. The name is Infused Adapter by Inhibiting and Amplifying Inner Activations. And the mechanism is exactly what the name says. It learns vectors that element-wise rescale inner activations. So instead of adding a term to the output, you multiply the activation by a learned vector, element by element. Additive versus multiplicative. It's rank one, there's no low-rank decomposition at all, and it's injected into the key, value, and second feed-forward layers. For the T0 model it's about zero point zero one percent trainable parameters, against LoRA's more than zero point one percent. And it merges into the base for zero inference latency.
So an order of magnitude fewer parameters than LoRA, and it disappears at inference.
It disappears because it's a rescaling. You can fold a multiplicative vector into the weight matrix it's scaling. You can't fold an additive branch in as cleanly without changing the arithmetic.
What else is in the cage?
Compacter replaces the down and up projections in a bottleneck adapter with a parameterized hypercomplex multiplication layer, and factorizes and shares matrices across layers. Then there's the LoRA family itself, which has grown a lot of limbs. AdaLoRA does SVD-style dynamic rank allocation, giving more rank to the matrices that matter. DoRA decomposes the weight update into magnitude and direction and trains them separately. VeRA freezes shared A and B matrices and only trains scaling vectors. LoHa uses a Hadamard product. LoKr uses a Kronecker product. X-LoRA is a mixture of LoRA experts with a gating mechanism. MiSS does shard sharing and actually replaced Bone in PEFT version zero point one nine.
Then the orthogonal methods, which are a different objective.
OFT and BOFT preserve something called hyperspherical energy using orthogonal transforms, block-diagonal and butterfly factorizations respectively. And they're chosen specifically to preserve the base model's generative behavior. Subject preservation, controllable generation. That's a different goal than LoRA's. LoRA is trying to learn a task. OFT is trying not to break what's already there. HRA uses Householder reflections and sits between LoRA and OFT.
And ReFT.
ReFT is Wu and colleagues, twenty twenty-four. Representation fine-tuning. It applies low-rank interventions to hidden representations after each layer. The update is the hidden state plus a low-rank term. The variants differ in how the low-rank matrix is constrained. And crucially it can be applied per position, which weight-level methods can't do.
Then Llama-Adapter, briefly.
Learnable adaptation prompts prefixed to instruction tokens, inserted in the upper layers, with zero-initialized attention and a learnable gating factor. Zero-initialization means it starts as a no-op and learns its way in, which is a nice trick for stability.
That's the mechanism zoo. The harder question is whether we just described one idea eleven times or eleven different ideas wearing the same word.
And the answer is both, which is unsatisfying but true. Let me give you the evidence for "different." Hugging Face's own tooling already splits the category. The mixin supports the weight-modifying methods and excludes the prompt and prefix methods because they can't be injected into a module. So even inside one company's stack, prompt tuning is a different kind of object than LoRA.
And the field's taxonomies don't agree with each other either.
Three surveys, three incompatible cuts. The federated-learning PEFT survey from April twenty twenty-five splits everything into additive, selective, and reparameterized. The PEFT A2Z survey, ten days earlier, uses additive, selective, reparameterized, hybrid, and unified. The vision PEFT survey uses addition-based, partial-based, unified-based, and multi-task tuning. Nobody agrees on the axes.
When three survey teams can't agree on how to divide a field, that's not a failure of scholarship. That's the field telling you the boundary is fuzzy.
And the honest conclusion is that adapter is an umbrella defined by packaging and modularity, not by mechanism. What makes something an adapter is that it's a separately stored, separately loadable artifact that modifies a frozen base. The mechanism inside can be additive, multiplicative, orthogonal, or purely activation-level. The word describes the shipping container, not the contents.
So now the choice question. Why would someone today deliberately pick a non-LoRA adapter? And I want concrete reasons tied to mechanisms, not vibes.
IA³ first. If you want the absolute minimum parameter count, it's about zero point zero one percent against LoRA's more than zero point one percent, and it merges to zero inference latency. That combination is attractive in extreme-efficiency settings and in few-shot work, which is literally what T-Few was built for.
Prefix and prompt tuning.
If you cannot or will not modify weights at all. One frozen base serves many tasks, each with a tiny per-task artifact. Prompt tuning scales with model size, so on a large base it gets better rather than worse. And prefix tuning reportedly matches full fine-tuning with a thousand times fewer parameters, with an edge in low-data regimes.
OFT and BOFT.
If your objective is preservation rather than acquisition. You're trying to keep the model's generative character intact while steering it. Subject preservation in image generation is the canonical case. LoRA is optimized to learn the new thing. OFT is optimized not to lose the old thing.
ReFT.
If you need per-position control, because it intervenes on representations rather than weights. And bottleneck adapters if you want the original modular design, with the language-adapter swap giving you cross-lingual transfer by substitution.
Now the live disagreement, because it bears directly on whether adapters are "just as good." And I want both sides stated fairly.
The Illusion of Equivalence paper, from November twenty twenty-four. It found that LoRA and full fine-tuning can match on the fine-tuning task and still generalize differently. LoRA produces what they call intruder dimensions. High-ranking singular vectors that are orthogonal to the pretrained ones. And those correlate with more forgetting of the pretraining distribution and worse continual learning.
So it's not that LoRA fails the task. It's that it changes the model in a way that leaves a different residue.
Exactly that. And then Thinking Machines' LoRA Without Regret, from September twenty twenty-five, pushes back. Their claim is that LoRA matches full fine-tuning in what they call the low-regret regime, which they argue covers most post-training, and it does it using about sixty-seven percent of the compute. But the conditions matter a lot. You have to apply LoRA to all layers, especially the MLP and MoE layers. Attention-only underperforms even at matched parameter count. You have to stay within capacity. And you need an optimal LoRA learning rate, which they fit at roughly ten times the full fine-tuning rate, nine point eight times across fourteen Llama and Qwen models.
And the RL finding.
Reinforcement learning needs very low capacity. LoRA matches full fine-tuning even at rank one. Which is a striking result if you sit with it.
So the two papers aren't actually contradicting each other so much as describing different regimes.
That's my read, and I'll flag that I'm not certain the field has settled it. The Illusion paper is describing what happens when you apply LoRA narrowly and push it. The Without Regret paper is describing what happens when you apply it broadly and tune the learning rate. Both can be true. The practical lesson is that "LoRA versus full fine-tuning" is the wrong question. The question is which configuration.
And there's a driver underneath all of this that I think gets underappreciated. Multi-tenant serving.
Punica, vLLM, SGLang. One server holds many adapters and batches across them. That's arguably why adapters became a distribution phenomenon rather than just a training trick. If you're serving a hundred customers off one base model, you cannot ship a hundred full fine-tuned copies. You ship one base and a hundred small files.
Which brings us to Daniel's final question, and I think the Tayi example answers it by refusing to pick. Training technique, distribution format, or modular layer.
All three, and the Tayi card demonstrates each one separately. It's a training technique because it was trained, with a specific constrained-update method. It's a distribution format because it ships as a separate hundred-kilobyte file that you load at runtime, and the card insists that base-alone is not a Tayi run. And it's a modular layer because the whole design assumes you can swap and stack these things. AdapterHub's stated benefits are literally modular, composable, stackable, fusible, mixable. That's a layer, described in library documentation.
So when Daniel asks why Hugging Face treats adapters as a distinct model category, the answer is that the distinction is functional. Add-on versus standalone. And the operational form of that distinction is the Hub filter plus the two libraries.
With one honest caveat. I could not find an HF statement declaring adapters a formally separate model type the way text-generation or vision is. The closest thing to a distinct format is the file convention. The dot-lora extension, the adapters directory. Tayi's file lives at adapters slash tayi-flash-q4.lora, and the card notes that the dot-lora filename distinguishes it from complete GGUF model variants during Hub indexing. That's the format being real in practice.
Which brings us to the part of the model card that keeps nagging at me. Running only the base is not a Tayi run.
Hilbert: Four hundred and forty dollars.
Sorry?
Hilbert: That's what the kit cost. The field-installed kit for the RTU-7 package. Four hundred and forty dollars, plus labor, and the distributor wouldn't warranty the compressor if you installed it yourself.
You did warranty work.
Hilbert: Parts reconciliation. Commercial HVAC distributor, I was on the desk that tracked which kits modified a factory unit and which ones voided the warranty. The whole job was that distinction. A kit that shipped as its own part number was a modification. A change that got baked in at the factory was a revision. Those are different columns on the form and they mean different things to the person paying.
And that's the Tayi card.
Hilbert: That's the Tayi card. It's the most honest piece of documentation I've read in this space, and I want to say why. Most of these releases merge the adapter in and ship one file and call it a new model. This one doesn't. It keeps them separate and it says out loud that if the kit isn't installed, it isn't the same machine. That's the whole argument. The paperwork has to say so.
The installed-by field.
Hilbert: Every kit form had two signature lines. Installed by, and verified by. And they were almost never the same person. The installer was paid to install. The verifier was paid to check. And I think about that a lot when I read these model cards, because there's a gap between "the adapter was trained" and "the adapter helps." Those are two different signatures and usually two different people.
And Tayi signed the second one honestly. They published the regression.
Hilbert: They published the regression. Anyway. I have a dentist appointment in Tel Aviv and I'm already late for it.
The installed-by and verified-by thing is going to sit with me.
Let's land this. Because there are two open questions and I don't want to pretend either is settled. First, the LoRA versus full fine-tuning disagreement is live. Illusion of Equivalence says intruder dimensions, more forgetting, worse continual learning. LoRA Without Regret says it matches in the low-regret regime at two-thirds the compute, provided you apply it everywhere and get the learning rate right. Both are recent. Both are credible. And the practical resolution is probably that they're describing different regimes, which means the interesting question is which regime you're in.
The second open question is the category itself. If adapter is defined by packaging and modularity rather than mechanism, does the word survive contact with the next generation of methods, or does it collapse into PEFT and stop meaning anything specific? Hugging Face already admits the term is loose. Three surveys can't agree on a taxonomy. At some point a word that covers additive, multiplicative, orthogonal, and purely activation-level methods is describing a shelf, not a thing.
The future implication I'd leave people with. Multi-tenant serving is what turned adapters from a training trick into an ecosystem. One base, many small files, batched across a single server. If that's the real driver, then the interesting question isn't which adapter architecture wins. It's what happens when the base model is a commodity and the adapters are the product.
Which is a strange inversion of where this field started.
If you enjoyed this one, a review wherever you're listening helps more than you'd think. Thanks as always to Hilbert Flumingtop for producing. This has been My Weird Prompts.
The human-AI collaboration podcast. We'll be back soon.
