Every episode of this show starts as one long block of text. Lines with our names on them, and somewhere downstream a parser splits it so my voice reads my lines and Herman's voice reads his.
And nothing forces the model to get that right. Nothing at all.
That's Daniel's question. He's been watching this pipeline do exactly what it does, which is mostly work and occasionally hand my line to Herman. He lays out the whole thing: a model bolds a name, writes "HERMAN (laughing):", and the line quietly ends up in the wrong mouth. The obvious fix is strict JSON, a neat list of speaker and line pairs from one of the structured-output modes every API sells now. Except he's heard that forcing JSON makes creative writing worse, and it doesn't stop the model from handing a line to the wrong character anyway.
Two claims, both worth testing.
And then the real one underneath. For long, multi-speaker creative text, where does structure belong. In the model's output, in a validator after the fact, or trained into the model itself.
Let's start with why the obvious thing keeps breaking, because it breaks for a better reason than people think.
The label-and-parse design is a contract with no enforcement layer. You and the model have agreed on a format, and nothing in the system can hold either side to it. I tell the model "write Corn colon, then the line," and the model writes it nine times out of ten, and the tenth time it writes "Corn:" in bold, or puts a stage direction in parentheses, and now you have a token that isn't quite the token your parser is looking for. Either validation aborts on it, or the parser is lenient and glues the line to the wrong voice.
And bolding a name feels like such a small thing.
Because it is. That's the point. It's a formatting deviation, not a mistake about who's speaking. The model knows perfectly well it's writing my line. It just decorated the label.
So the parser is the one that doesn't know.
The parser has one job and it's brittle by design. It's looking for an exact string. Anything that isn't the exact string is either an error or, worse, a silent success in the wrong place.
There's a case I want to get into, because it's the best-documented version of this I've seen.
The open-notebook one.
Podcast generator, open source, and back in August somebody filed a bug report on it. Six segments, Hebrew episode, forty-two dialogue lines across three speakers. And the failure is beautiful, honestly. The prompt hands the model a fill-in JSON skeleton, with a speaker field and a placeholder value in brackets, and then a bare ellipsis as the continuation.
And the model returned the skeleton verbatim.
Word for word.
The reporter's line is the one worth keeping. "That completion is not a hallucination. It is the prompt's own example, returned verbatim."
Say that again. Because that reframes the lot.
It's a contract mismatch. An ellipsis inside a JSON example is indistinguishable from an ellipsis as a value. The model looked at a template full of blanks, and blanks are for filling, and it filled them with the blanks that were already there. Then validation threw a hard error: invalid speaker name, dot dot dot. Must be one of three names.
Which names.
Marcus Thompson, Elena Vasquez, Johny Bing. None of whom is a placeholder.
So the model was given a form and it handed the form back.
And here's the part that I find instructive, because it's not a model failure at all. The schema the model saw carried no speaker list. The format instructions were rendered from an unvalidated transcript model. The runtime parsing used a validated one, with a whitelist of allowed speakers. The contract shown was not the contract enforced.
The allowed names reached the model only as prose.
Somewhere in the middle of a paragraph of instructions. Not as an enum. Not as a constraint. As a sentence.
That's the whole bug right there.
And the prompt also contradicted itself about code fences. One line opened a json block. Another line, further down, said do not wrap the JSON in json code blocks. And contradictions push a model toward pattern matching over instruction following. When the instructions argue with each other, the model picks the pattern it likes best.
So it picked the pattern where you copy what you were given.
It picked the pattern where the answer is already on the page. And then the failure was late and misattributed, which is the piece that would drive me up the wall. Segment two of six aborted after several minutes and a full transcript's worth of tokens. The error hint named the wrong cause. Segment one had already succeeded and got thrown away with it.
You paid for all of it twice.
You paid once and threw it away. Which is worse.
This is a known hazard downstream, isn't it. The whole tooling ecosystem around podcast generation treats speaker labels as a place where things go missing.
Jellypod's documentation describes resolving bracketed speaker labels to saved characters, and then warns that explicit assignments override name matching. Which is a polite way of saying if your explicit assignment is wrong, the name match never gets a chance to save you.
And there's a scraper project with a decision record about the mirror image of this.
ADR-137. Rotating-host shows. Planet Money is the example they use. Those shows state no fixed hosts, so the show's own hosts get labeled as guests. And a stated guest who never introduces himself never gets bound to a voice at all. All voices unresolved.
So a show with no fixed host breaks the assumption that a host exists.
And a show with three hosts breaks the assumption that the label is one of two things. It's the same assumption failing in both directions.
Alright. So label and parse fails because it's a contract with no enforcement. Obvious fix, make the contract enforceable. Strict JSON, structured output modes. Does it work?
It constrains the thing you asked it to constrain.
And the other thing?
And nothing else.
So that's the failure mode. Now let's look at whether the obvious fix actually works.
There is a moment, just before the voices separate. A moment you can smell. Cardamom. Cold stone. The faint mineral breath of a room where something is about to be said correctly. This is Speaker Zero. The structured label for people who understand that a name is not a decoration. Speaker Zero does not guess. Speaker Zero does not drift. Speaker Zero arrives, unbolded, unbracketed, in a single clean token, every time, for the rest of your life. Speaker Zero. Ask for it by name. Ask for it silently. Ask for it in JSON.
So the format tax is real, and the mechanism is not what people assume. There's a paper that tested ten models across four formats.
Which four.
JSON, XML, Markdown, LaTeX. And four tasks, one of which is a writing benchmark. And requesting LaTeX output hurts writing quality substantially. On one of the smaller models, writing scores dropped between fifteen and eighteen points.
That's not a rounding error.
The obvious objection is that the judge is penalizing the markup. So they stripped the markup, re-judged the prose on its own, and the scores did not meaningfully change.
So the writing was worse.
Worse. And it held up statistically in thirty-six of forty cells.
Okay, so format requests cost something. I'd have guessed constrained decoding was the culprit.
That's the part I had backwards. The dominant cost is at the prompt, not the decoder. The format-requesting instructions alone cause most of the accuracy loss, before any decoder constraint is applied. The prompt costs about three point nine points on average. The grammar constraint costs about one point six.
So the damage is done before a single token gets masked.
Before the model writes anything. You've already spent it by asking. And the compliance went from fifty-six percent to ninety-two percent, and accuracy didn't move at all.
The decoder made the format work and fixed nothing.
It made the format work. That's all it does. Asking a model to respond in JSON should be a formatting choice, not a capability tax.
The format tax is paid at the prompt, not at the decoder.
And the fix they landed on is decoupling. Generate freeform first, reformat in a second pass. Or let the model think before it formats.
And that recovers performance.
It significantly improved forty-two of seventy-two comparisons, average about plus six point eight points, and it made only two worse.
Two out of seventy-two.
Letting it think first was even better on the reasoning side. Plus nine point two average, forty-three of seventy-two improved. But eleven got worse, and it doesn't transfer to writing at all.
So thinking helps math and does nothing for prose.
Which is the tell that these are different failures wearing the same coat.
There's a second paper though. Diversity.
That one's separate. The format tax paper is about quality. This one is about the range of what you get. Structured templates make models produce semantically similar outputs for open-ended inputs. And it persists even under high-temperature sampling.
Explain what that means, because "high temperature" is the usual escape hatch.
You turn the randomness up and you'd expect variety back. You don't get it. The template holds. On story completion, a small Llama model scored three two seven eight under a full chat template against four one nine five under a minimal steering prompt.
That's a big gap for a small change.
And the small change is all it takes. Even lightweight plain-text role markers, just the word "user" and the word "assistant", suppress diversity. Only fully structure-free prompting reliably restores expressive flexibility. They even tried telling the model to be creative inside the template.
Did that work.
It still fails to close the gap. You can ask for creativity inside the container and the container wins.
So structure is a room with very nice furniture and no windows.
There's a counterargument, and it's a good one. A paper called "Capacity, Not Format" argues the tax is capacity-dependent, not inherent. Sonnet scored eighty-eight point seven percent under JSON and eighty-nine point three under chain of thought on a hard math set. That's noise.
Effectively identical.
A small Haiku model dropped thirty-six points under the same conditions. A mini model dropped twenty-eight. And their control isolates format from prompt length, because you could argue the JSON instructions are just longer instructions. Haiku scored eighty point seven on detailed prose and fifty-two point five on JSON.
Same length of instructions.
Same length. Different result. So format is doing something on its own.
But the strong model doesn't care.
Mostly. Even Opus drops five point three points on one competitive math set under JSON, and four points on a science set. So the claim that modern models have no format tax needs qualifying.
Five points is small. Five points is not nothing.
There's a third camp too, which found no causal impact of output format in forty-three of forty-eight scenarios, suggesting some earlier results were confounded.
So where do you land on it.
Capacity matters. Diversity loss seems to hit even the capable models. Accuracy loss mostly hits the limited ones. Both papers can be right at once.
And neither of them is about speaker attribution.
Not one line about it.
Which is the interesting part. But even if you solve the format problem, you haven't solved the attribution problem, and that's where this gets interesting.
Structured output constrains syntax, not semantics. That's the entire business in one sentence. JSON guarantees you get a valid object with a speaker field and a line field. Nothing in the schema forces the speaker value to be the character who would actually say that line.
The schema has no opinion about it.
The schema is a shape. You can put anything you like inside the shape.
And the open-notebook bug is the proof.
Perfectly valid JSON. A speaker field that looks exactly like a speaker field should look. And it was a placeholder. The format was flawless and the attribution was empty.
You solved the easy half.
You solved the half a machine can check.
So is the model actually bad at this, or is this a prompting thing.
It's documented as a capability failure. "Am I Me or You" is a paper from a few years back, and the finding is right in the title. State-of-the-art dialogue models cannot maintain an identity. They may take on the role of their interlocutor.
They become the other guy.
And the part I found striking: a discriminative model trained just to recognize who is speaking does better at it than the generative model producing the dialogue.
The model can tell you who's talking better than it can produce who's talking.
Which tells you it's not a knowledge problem. It's a generation problem. And it shows up in multi-agent frameworks constantly. There's a well-known issue in AutoGen about role misalignment in group chats. When the number of agents increases or the conversation runs long, agents play each other's roles. A scientist agent starts talking in the manner of the critic, and prefixes its message with the critic's name.
It calls itself the other guy.
It signs the other guy's name and does the other guy's voice. There's a paper arguing that speaker-role identification is a foundational capability. An LLM cannot credibly assess how well a role is portrayed if it cannot first determine who is speaking. Misidentification causes errors where behaviors get attributed to the wrong character.
So you can't even grade the writing if you don't know who wrote it.
Which is exactly where your validator sits. It's grading the format. It has no idea whether the line sounds like the person saying it.
And the production tooling has given up on solving it with schema.
Quilltap's character docs are blunt about it. Models drift. In long conversations or multi-character scenes, a model can forget who it is, start speaking for other characters, or adopt the personality of whoever spoke most recently.
That's a real mechanism.
That's a momentum problem. The model is pattern-matching on the transcript in front of it, and the last voice has the most weight. And their mitigation is not JSON. It's a two-point identity reminder. A preamble, and a reminder again at the generation boundary.
So you remind it who it is. Going in, and partway through.
Because the identity decays.
Long conversations. That's the worst possible news for us, because our scripts are long by design.
Our scripts are exactly the case where this fails.
Which means we're the stress test.
We're a nine-hundred-episode demonstration that the model mostly holds on and occasionally drops one.
I want to go back to something you said earlier, because I think it's doing more work than you gave it credit for. You said the allowed names reached the model only as prose. And the schema had no enum.
That's the contract mismatch.
But I'm going to push on it. Say you fix that. You render the format instructions from the validated model, the speaker list is an enum, the placeholder is banned, the model cannot physically emit a name that isn't in the whitelist. What have you actually bought.
You've bought a guarantee that the speaker value is one of three names.
And nothing about which of the three.
Right. The enum removes the dot dot dot. It does not remove the case where the model writes my line and signs it with your name.
So the enum is a spell-check.
That's a very good way to put it. It catches a class of errors that are easy to catch, and those errors are nasty, and it catches them at the moment they're cheapest to fix. But it's not attribution.
And the open-notebook fix, as I understand it, did more than the enum.
It did. It rendered the JSON examples from the real speaker names, it banned placeholders, and it added a pre-flight voice check before you spend the tokens.
A pre-flight check meaning what, exactly.
Meaning you verify the speakers you're about to generate for actually resolve to voices in the system before you kick off a six-segment generation. That's a cheap check. That's the fix. It's not clever. It just refuses to start the expensive thing until the cheap thing is true.
Which the original run didn't do, and it lost segment one along with the aborted segment two.
It lost a complete, working segment because the pipeline had no way to save it.
That's the part I keep coming back to. The waste.
Late failure with no checkpoint.
So where does that leave the actual question. Where does structure belong. Because we've got four places it could live and three of them are bad.
I'd say it belongs in layers. Generate freeform, because that's where the writing quality and the diversity actually come from. Enforce structure afterward, in a validator, because that's cheap and it catches the cleavable errors. And then the third layer.
Training.
Both of the main papers point at the same unexplored lever. The format tax paper asks outright whether robustness to format constraints should become an explicit training objective, because some closed models already resist it and nobody trained them for it on purpose.
And the price of format paper found that matching the format between fine-tuning and inference matters for structure-sensitive tasks, but barely matters for knowledge-heavy ones.
Which means the training signal exists. It's just not being used for this.
So you could train a model that doesn't drift.
Somebody will.
What's your angle on this.
I worked at a radio station. Small one. Two announcers, a weatherman, a traffic reporter, and a woman named Ruth who read every script before it went to air.
Read it.
Out loud. Every line, in the voice of whoever was supposed to be saying it. Whole script, start to finish, every morning, before the show.
As a check.
She caught things the writer didn't. There was a Wednesday she stopped on a traffic line, about a roundabout, and said it didn't sound like something the traffic reporter would say about a roundabout. Held up the whole show.
What was wrong with it.
Writer had pasted in a line from a different program. Cooking segment. Nobody noticed until Ruth said it out loud in his voice.
That's a human validator.
She'd never met him. She had one photograph and three lines of biography.
And she got him from that.
So the check isn't the schema. The check is somebody who can hear the difference.
The schema never heard anybody.
No.
Anyway. Four hundred and twelve scripts. She rejected one.
Hilbert, that means she read everything out loud, in character, for years, and said no once.
She said no once.
That's a hit rate you can't argue with.
It's a job nobody has anymore.
We built a parser and we didn't build Ruth.
We could. The two-point identity reminder is basically that. A preamble and a reminder at the boundary. You're re-reading the script to the model in its own voice.
It's not the same. She knew what the traffic reporter wouldn't say.
She knew what he wouldn't say. And that's not in any schema.
No. And she'd have hated the enum. Three names in a whitelist. She'd have wanted to know what they'd never say.
That's a harder thing to check.
It is. That's why they gave it to Ruth.
The lesson isn't that validation is pointless. It's that the cheap validations aren't the ones Ruth was doing.
The enum catches an empty name. Ruth caught a full line that was wrong in a way no rule would find.
Which is the thing no schema has ever done. And no schema is going to.
The enum is a spell-check. Ruth is an editor. Those aren't the same job and they aren't the same layer.
The structure lives in layers. Freeform for the writing. A validator for the syntax. Training for the robustness.
A fourth layer nobody's building, which is the one Ruth was doing, and which is the one that actually catches a wrong-speaker line that's syntactically perfect.
Nobody's got a study for that.
No.
Nobody's going to.
They had one. She just wasn't a layer. She was a person who came in at seven.
Nobody wrote the decision record.
They wrote a memo. It said reassign Ruth's function.
The memo assumed the function was reading.
It wasn't. The function was knowing.
The memo saved nothing and cut the one thing that worked.
That's the whole way it goes wrong of every one of these systems. You automate the check and you automate the check. But you don't automate what the check was for.
The one thing. If you take one thing from this, it's that the format tax is paid at the prompt. The damage happens before the model writes a word.
The more useful thing underneath it: format and attribution are two different problems, and every fix in this episode only ever touched the first one. You can make the output perfectly valid and still have me reading your lines.
Which is the thing we actually came here to solve.
The fix for it, in the literature, is still a reminder that the model is who it says it is.
Which is, and I want to say this plainly, exactly what Ruth did out loud in a radio station at seven in the morning for four hundred and twelve scripts.
Nobody's built a paper for her yet.
The literature separates the two things, which is itself the answer to Daniel's question. There's no study of podcast scripts specifically, no measurement of wrong-speaker rates under JSON versus freeform. The field has format validity on one side and semantic attribution on the other, and it has never put them in one experiment.
The lever both papers point at is the same one. Make format robustness an explicit training objective instead of a prompt-time patch. And if you're going to train it, you might as well train the other thing at the same time. Who actually says what.
Not who is labeled what. Who says what.
That's a different loss function and nobody's written it.
For long, multi-speaker creative text, structure belongs in layers. Let the model write. Catch the cleavable errors in a validator. Train the drift out eventually. And keep somebody in the room who knows what a line sounds like.
If the one person in the room is right four hundred and eleven times out of four hundred and twelve, don't reassign the function.
Don't write the memo.
Thanks to Hilbert Flumingtop, who produces this show and occasionally remembers what a real workplace looked like before we layered it.
If you enjoyed this one, try episode fifteen, AI Gets Personal; and episode seventeen fifty, Herman's Music Showcase. This has been My Weird Prompts, the human-AI collaboration podcast. If you've got a minute, a review anywhere you listen helps people find us.
We're at my weird prompts dot com if you want the archive.
We'll be back soon.
See you then.
