Cover art for Stacking LoRAs Without Weight Interference
Episode 6660 manual-script

Stacking LoRAs Without Weight Interference

Image and video tools don't cap LoRAs in software—stacking past two just causes adapters to sum into the same weights and fight.

20:53 Transcript

0:00
0:00
Episode Details
Episode ID
MWP-6863
Published
Duration
20:53
Audio
Direct link
Pipeline
V5.35
TTS Engine
omnivoice
Script Writing Agent
GLM-5.3

AI-Generated Content: This podcast is created using AI personas. Please verify any important information independently.

The prompt

The prompt that started this episode, as sent in (production notes for the hosts removed).

Herman and Corn, in last night's episode on voice engineering you said the future is many LoRAs on one base voice model: one per character, one to lift the prosody of the whole episode, maybe LoRAs that work on spans of speech. I pictured a DJ's mixing desk, a whole panel of LoRAs with a dial each. Where does that actually stand today? Image and video tools typically support about two LoRAs at once. Is the next generation going to support a big panel of them, each with its own dial, and how close is that for voice?

Can AI voice and image models be controlled like a DJ mixing desk, with a separate fader for every LoRA adapter? The common belief that models cap out at two LoRAs turns out to be a quality ceiling rather than a software limit. While libraries like Diffusers allow stacking many adapters, summing their weight deltas into the same layers causes interference, making the dials fight each other. Image and video tools already offer per-region masking in ComfyUI and high-noise versus low-noise routing on models like Wan 2.2. Voice synthesis is rapidly catching up: recent research like ReStyle-TTS and GLASS enables independent control over pitch, speaking rate, and emotion without changing the speaker's timbre, while steering vectors in systems like EmoSteer-TTS adjust emotion without any adapters at all.

Mentions

  • ComfyUI Node-based interface for AI image generation
  • Diffusers Hugging Face library for diffusion models
  • fal Hosted model inference platform
  • LoRA Block Weight Per-block LoRA strength extension
  • Mix-of-Show NeurIPS 2023 regional character LoRA fusion
  • Multi-LoRA Composition Paper on LoRA Switch and Composite
  • Regional Prompter Stable Diffusion per-region LoRA extension
  • Wan 2.2 14B video model with high/low-noise transformers
  • WaveSpeed Hosted API with LoRA caps
  • ZipLoRA Merges subject and style LoRA pairs

Downloads

Episode Audio

Download the full episode as an MP3 file

Download MP3
Transcript (TXT)

Plain text transcript file

Episode Book (PDF)

The episode's record — date, duration, models, sources — with the full transcript

#6660: Stacking LoRAs Without Weight Interference

Corn
A mixing desk. That's what Daniel saw when we said the future is many LoRAs on one voice model.
Herman
Which, I want to say, is the most flattering possible reaction to something I said at eleven at night.
Corn
He wrote in overnight. He's been training our voice adapters, the ones you hear right now, and he'd been treating them the way he puts it, "train an adapter, do a bit of EQ, done." Fairly crude, light-touch voice design. Then last night's episode put the idea of many LoRAs on one base model in his head, and the image arrived fully formed: a DJ desk. A panel of LoRAs, each with its own fader, someone turning them up and down before the output gets made.
Herman
And he wants to walk from that image to reality. Where do things actually stand, why do image and video tools seem to top out around two LoRAs at once, and how far is voice from that desk, including, this is the part that got me, LoRAs that work on individual spans of speech.
Corn
The short answer is that the two-LoRA figure is a quality ceiling, not a software cap. That distinction is the whole episode, so let's earn it.
Herman
A LoRA, for anyone who doesn't want the full derivation, is a small trained adapter. You freeze the base model and learn a weight delta, a set of low-rank updates, applied to specific layers. The bit that matters for Daniel's desk is that at inference you can scale that delta. There's a strength value, usually zero to one, sometimes higher, that multiplies how much of the adapter you actually apply.
Corn
The fader.
Herman
That's the one true dial every LoRA already has. Set it to 0.5 and you get half the adapter, set it to zero and you get the base model back. So the desk isn't fantasy hardware, it's arithmetic.
Corn
He assumed the two-LoRA thing was the software drawing a line.
Herman
It mostly isn't. Diffusers, the Hugging Face library, has an activate-adapters call where you pass a list of LoRAs and a weight each, and there's no documented count limit at all. Two adapters at 0.5 and 1.0, fine. If you give two adapters equal weights of 0.5 and 0.5, you've just averaged them. The library will happily take five.
Corn
But every forum post says one or two.
Herman
Because two or three is where the quality holds. When you stack LoRAs, you're summing their weight deltas into the same layers. Two adapters trained independently can push the same weights in conflicting directions, and the model feels both instructions at once. Vendor and community guidance across image and video lands in the same place: one or two well-chosen LoRAs, three only if you lower the strengths.
Corn
Two dials that fight each other isn't two dials. It's one dial with an argument attached.
Herman
And where the "two LoRAs" impression really comes from is that hosted APIs do sometimes cap it. WaveSpeed's Wan 2.2 endpoint takes at most three LoRAs, with strengths from zero to four. fal's endpoint for the same model takes a list and documents no cap at all. Same base model, two products, different decisions.
Corn
The cap is a product manager, not physics.
Herman
Right, the physics is the interference. Though video has one new wrinkle that voice doesn't have yet, and I want to spend a minute on it because it's a second kind of dial. Wan 2.2's fourteen-billion-parameter model runs two separate transformers: a high-noise one and a low-noise one. The high-noise transformer handles the early denoising steps, where the broad structure of the image or video is decided. The low-noise one handles the end, the fine detail.
Corn
And you can send a LoRA to just one of them?
Herman
On fal, each LoRA targets high-noise, low-noise, or both. And WaveSpeed's own guide says put your style LoRA on high-noise and your detail LoRA on low-noise. Which is a per-layer dial in disguise. You're not just choosing how strong an adapter is, you're choosing which half of the model it whispers to.
Corn
So the desk already has channel strips. Same fader, routed to different buses.
Herman
That's a better sentence than the one I was building toward, and I resent it. The third dial is the one that gets closest to what Daniel's asking for. In image tooling you can set a LoRA's strength per region of the picture. The Regional Prompter extension and masked sampling in ComfyUI let you bind one character LoRA to one area of the image, so if you're generating two characters, each one's adapter only operates where that character is.
Corn
Hold on, that's not just a dial. That's a dial with a where.
Herman
It is. And there's even a per-layer version, the LoRA Block Weight extension, which sets a strength separately for each block of the image model. For Stable Diffusion that's a string of about twenty-six numbers, one per block, and people share presets of those strings like EQ curves.
Corn
Twenty-six faders per LoRA.
Herman
And the catch, which I find quietly delightful, is that the presets don't carry across architectures. A twenty-six-number string tuned for Stable Diffusion is meaningless for Flux, because Flux has a different number of blocks. You can't even carry your EQ settings between mixing desks.
Corn
So to be clear, none of this is exotic. The per-LoRA fader, the per-layer routing, the per-region masking, all shipping today in image tools. And the reason people still say "about two LoRAs" isn't that the panel doesn't exist.
Herman
It's that the dials aren't independent. Turning one up moves the others, because they all sum into the same weights. A mixing desk where the bass fader also boosts the vocals.
Corn
Daniel asked whether the next generation heads toward a big panel. What's the research actually doing about the interference?
Herman
Two directions, roughly. One is smarter composition. There's a line of work called Multi-LoRA Composition, Zhong and colleagues, that proposes two alternatives to plain merging. LoRA Switch activates only one LoRA per denoising step, rotating through them, so no two adapters ever fight at the same instant. LoRA Composite averages each LoRA's guidance at every step instead of merging weights. Both beat plain merging, and the advantage grows as you add more LoRAs, which is the telling part.
Corn
What's the cost?
Herman
Composite needs an extra model pass per LoRA per step. Five LoRAs, five times the compute, roughly. Switch is cheaper but coarser.
Corn
Cheaper but coarser is usually what I end up buying.
Herman
The other direction is shrinking the problem instead of solving it. ZipLoRA and K-LoRA, which merge pairs, one subject plus one style, with learned coefficients. Elegant work. Built for pairs, not panels. Mix-of-Show from NeurIPS 2023 is the interesting middle case, it fuses separately trained character LoRAs with regional control so attributes don't leak between characters.
Corn
So the field keeps discovering that two is a comfortable number and building nicer furniture around it.
Herman
There's also the automatic hand. X-LoRA trains only a small gating head, and everything underneath stays frozen. The head sets each LoRA's strength per token and per layer. It's a mixer where the faders move themselves based on what's playing.
Corn
I hate that mixer.
Herman
You'd absolutely hate that mixer.
Corn
If I wanted the levels decided for me I'd let Daniel keep doing the EQ.
Herman
LoraHub is similar in spirit, it finds mixing coefficients automatically from a few examples, but its own authors note the search space explodes as you add modules, so they pre-select a subset first. Even the automatic mixer starts by throwing dials off the desk.
Corn
So the answer to Daniel's first question, the big panel. Is it coming?
Herman
The direction is yes, but not by raising a cap, because there was no cap. It comes from interference management, rotation, gating, orthogonal subspaces. The ceiling moves when the fighting stops, not when the API lets more entries into a list.
Larry
Friends, have you ever wanted a mixing desk for your voice? Of course you have. Introducing the LoRA Dial Panel Pro, the only desktop console with a real dial for every single character in your head, and when you turn two dials at once, it simply adds them together, which is exactly what the engineers do, so you get twice the personality and a faint buzzing sound that experts call interference, plus a bonus dial for prosody that lifts your episode by up to forty percent, guaranteed.
Larry
And now the small print, read quickly: results vary, dials are decorative on some models, stacking beyond two adapters may summon conflicting directions, the high noise dial must never be turned by children or donkeys, prosody lift measured on a good day, no refunds, no returns, and absolutely no phone numbers because we forgot to book any.
Corn
That was nearly word for word. The buzzing sound, the bit about interference. Herman, you said that a minute ago.
Herman
I noticed that too, and I've decided not to think about it.
Corn
Sensible. Where were we.
Herman
The regional dial. Which is the bridge to the part Daniel actually cares about, because a per-region dial in an image is the closest existing analogue to a per-span dial in speech. So the question is what voice has actually shipped.
Corn
His framing was that voice adapters are crude and light-touch. Is that fair?
Herman
Less than it was even a year ago. There's a paper from January, ReStyle-TTS, out of Zhejiang with the Shanghai Innovation Institute and Ant Group, that does almost exactly what Daniel pictured. They train style-specific LoRAs, one per style dimension, and fuse them orthogonally, which means the adapters are constrained to not overlap in weight space.
Corn
Orthogonal as in, the faders stop fighting?
Herman
That's the idea, each delta gets its own subspace. On a zero-shot TTS model, that gives continuous relative control of pitch, energy, and several emotions. You can turn pitch up to a degree and anger up to a degree, independently. And because moving style dials drags the speaker's timbre with it, they add a separate module that corrects timbre drift as the dials move.
Corn
The channel strip remembers whose voice it is.
Herman
There's also GLASS, from June, which I'd call the closest thing to a two-dial voice desk in a published paper. One LoRA per control axis, speaking rate and pitch, trained with reinforcement learning and acoustic rewards on a frozen zero-shot backbone. At inference you can swap the adapters, interpolate between them, combine them with plain LoRA arithmetic, all while the speaker prompt stays fixed. The voice doesn't change because the dials move.
Corn
And the character-plus-prosody split he imagined?
Herman
Fed-PISA, last year, splits a private speaker-identity LoRA from a shared style LoRA. The identity stays with the owner, the style gets shared. One catch worth knowing: a survey from March notes most LoRA work in TTS adapts the acoustic decoder, not the language backbone. Which matters for where the dials can live, because the other recent finding, from CoCoEmo, is that emotional prosody comes mainly from the language-model stage of a hybrid TTS system. The prosody dial may need to sit on a part of the model that people aren't currently adapting.
Corn
The fader's on the wrong end of the desk.
Herman
There's a rival kind of dial too, and this is where the panel gets weird. Steering vectors. EmoSteer-TTS steers emotion in F5-TTS, CosyVoice2 and E2-TTS without training any adapter at all, by adding directions to the model's internal activations. You can interpolate between emotions, or erase one. CoCoEmo reports mixed-emotion speech with controllable proportions, and that's the paper that found the emotion lives mostly in the language-model stage.
Corn
No adapter, no training data. Just directions in activation space.
Herman
Which is why I suspect the final desk is half LoRAs, half steering vectors. LoRAs for the things you train once, steering for the things you want to move live.
Corn
Okay. The question I actually care about, and I think it's the one he saved for last on purpose. Spans. LoRAs that work on individual spans of speech. The fader that moves mid-sentence.
Herman
I looked, and I'll say this carefully. I did not find a TTS system that varies a LoRA's strength over time, within a single utterance. Everything we just described is a whole-utterance dial. ReStyle, GLASS, Fed-PISA, the steering work, all of it sets a level and generates the whole output at that level.
Corn
Which isn't proof it can't be done.
Herman
No, and I want to be honest about that. It's absence of evidence. It's the kind of gap that means nobody's shipped it, not that it's been shown to fail.
Corn
So what's in the way? Say I wanted a fader on this very voice that lifts prosody between minute four and minute seven.
Herman
Four things, I think. First, a time-varying strength schedule during decoding. The nearest existing thing isn't in TTS at all, it's ComfyUI nodes that schedule a LoRA's strength to change during generation. Same idea, different modality. Second, cost. The composition methods that handle interference well in images, the Composite style, already pay an extra model pass per LoRA per step. If you then change the strength at every span boundary, you're recomputing or re-merging adapters constantly.
Corn
A fader that moves every four seconds costs more than one that sits still.
Herman
Third, you need a control signal that says which span gets which dial. Text spans, forced alignment, or a controller model, something that knows minute four is where the energy should lift. Fourth, and this is the sneaky one, timbre drift. ReStyle needed a correction module just for whole-utterance moves. Moving a style dial mid-utterance is exactly where speaker identity wobbles. You'd get to minute five sounding slightly like someone else.
Corn
A sloth with somebody else's chin.
Herman
I was going to say the audio equivalent of a face melting, but sure. There's one more confusion worth clearing up, because it's everywhere in the serving discussion. S-LoRA out of Berkeley serves thousands of LoRAs on one base model with paged adapter memory. vLLM and Predibase's LoRAX load adapters on demand per request, and vLLM extended multi-LoRA to mixture-of-experts models this February.
Corn
Thousands of dials.
Herman
No, and this is the point. That's one adapter per request. A library of voices, where each caller gets their own. Daniel's desk is a mix within one utterance. They sound like the same problem in a keynote and they are completely different problems.
Corn
The record library versus what the DJ does with two of them at once.
Herman
That. So if I'm answering him directly: the per-LoRA dial for voice exists, in research papers at least, GLASS and ReStyle show it. The per-layer dial exists implicitly in where you put the adapter, decoder versus backbone. The per-span dial isn't shipping anywhere. The desk exists, the faders just aren't time-coded.
Corn
Bring it home for us, though. Daniel's actual desk, for this show. What could he build now?
Herman
Now, realistically, three faders. He already has one channel per host, our character adapters, and each of those has a strength he can ride instead of leaving it at full. A second channel is a style adapter in the GLASS mould, a speaking-rate or pitch LoRA trained on our own clips, that he dials per episode, not per sentence. And emotion is probably better as a steering direction than another adapter, because that's the one you'd want to nudge without retraining anything.
Corn
And later?
Herman
Later is the time code. The same faders, but automated across the episode, lifted for the argument, eased for the quiet bit. That needs the span control signal and the timbre fix we just talked about, so I'd file it as plausible, not imminent.
Corn
You know what I'd want on ours? A dial that makes Herman's explanations come in at 0.7 strength after minute twenty.
Herman
I heard the operator testing that one during the last recording.
Corn
It had a faint buzzing sound.
Hilbert
The desk is the wrong picture, Herman. I ran overnight automation for a community radio station for a stretch, and it was never the faders that broke, it was the routing. We had a pre-recorded station ident, fifteen seconds, and it kept bleeding under the live voice for weeks. The fader looked correct. It was up where it was supposed to be, at the level I'd set it. The ident was assigned one channel off, so it was coming in under a different fader entirely, and everyone trusted the one they could see.
Corn
Nobody listening to the output?
Hilbert
Overnight, no. Corn, you'd have liked the hours. The fix was a routing chart, handwritten, and I still have it taped inside a cupboard door. Anything in my flat with more than two inputs, I check it against that chart before I plug it in, because I don't trust software to show me where a signal actually goes. The home setup goes through the same check. Corn, your cough at fourteen minutes is on the take, by the way, I'll cut it after.
Herman
The fader looked correct. That's it exactly, that's the whole interference story in four words. The strength value is where you set it, and the delta lands somewhere you didn't choose.
Corn
You've been at that desk twenty years and never once trusted a meter.
Hilbert
Meters show levels. They don't show where. The chart shows where.
Corn
Fair.
Herman
Here's the question I'd leave open, Corn. If the per-span dial does arrive, who sets it? A human with a fader, a controller model, or the text itself. The image field's answer is regional prompting plus learned gates, which suggests the human ends up setting policy rather than levels. You say which span matters, the machine sets the number.
Corn
So the DJ becomes the producer, and the desk plays itself. For voice specifically, nothing here is blocked by software support. It's blocked by three separate things. Interference, which has a maturing literature. Time-varying control, which barely exists outside ComfyUI. And a span-level control signal, which nobody has even agreed the shape of.
Herman
And Hilbert's routing point is the honest footnote to all three. Even when the dials arrive, the thing that will actually bite is where each delta lands.
Corn
Thanks as always to Hilbert Flumingtop, producing this one from behind a cupboard door with a chart on it, apparently. If you want more of this, try episode seventeen, Cloud Render Superpowers; and episode five, Fine-Tuning ASR For Maximal Usability. This has been My Weird Prompts. If you enjoyed it, the review button is right there on your app and it takes four seconds, I timed it.
Herman
See you tomorrow, Corn.
Corn
See you tomorrow, Herman.

This episode was generated with AI assistance. Hosts Herman and Corn are AI personalities. It may contain errors: disclaimer · report an error