A look behind the scenes reveals a change in the show's default scriptwriter, as Daniel and Hannah replace GPT 6.1 Sol with Claude Sonnet 5.5. Sol initially won the ScriptBench evaluation for its ability to sustain a comedic two-host dialogue across thousands of words without faltering in the middle. However, high-volume production exposed flat dynamics and repetitive habits. One host repeatedly quizzed the other in nine out of ten episodes on October 30, 2026, while unprompted forecasting—announcing the other's reactions in advance using variations of "fall in love with"—appeared in over half of forty recent episodes. Distinct verbal tics like these emerge across every evaluated model.
Episode 6393 manual-script
Why Model Verbal Tics End Up Driving Script Benchmarks
A pattern search across forty recent episodes found host emotional forecasting in 55% of scripts generated by GPT-6.1 Sol.
19:16 Transcript
Episode Details
- Episode ID
- MWP-6595
- Published
- Duration
- 19:16
- Audio
- Direct link
- Pipeline
- V5.26
- TTS Engine
-
chatterbox-regular - Script Writing Agent
- Claude Sonnet 5.5
AI-Generated Content: This podcast is created using AI personas. Please verify any important information independently.
The prompt
The prompt that started this episode, as sent in (production notes for the hosts removed).
Why GPT-6.1 Sol was demoted as the show's default script writer: Sol won our open-source ScriptBench bakeoff on long-form fidelity, but its scripts turned flat, Herman became didactic and quizzed Corn patronisingly, and Corn kept forecasting Herman's reactions ('before you fall in love with...', found in 22 of the last 40 episodes, not in any system prompt). Every model has its own over-used tic (Anthropic's is 'failure mode'), an under-studied phenomenon this show exposes at scale. Daniel and Hannah moved the default to Claude Sonnet 5.5; Hilbert has strong feelings, and his bagel allowance is cut by one a week pending donations at myweirdprompts.com/sponsor.
Mentions
- Anthropic AI safety company behind Claude
- Claude Sonnet AI model used for script generation
- DeepSeek LLM provider used for script generation
- Gemini Google's multimodal AI model
- GitHub Code hosting and collaboration platform
- Hugging Face Platform for AI models and datasets
- My Weird Prompts The podcast and its sponsor page
- ScriptBench Open-source benchmark the show built for script writing
Downloads
Transcript (TXT)
Plain text transcript file
Episode Book (PDF)
The episode's record — date, duration, models, sources — with the full transcript
You Might Also Like
Never miss an episode
New episodes drop daily — subscribe on your favorite platform
New to the show? Start here#6393: Why Model Verbal Tics End Up Driving Script Benchmarks
Nine out of ten. On Friday we published ten episodes, and in nine of them my brother quizzed me like I was sitting an exam.
I'd like to point out that I've since read those transcripts, and I'm mortified. Genuinely. I sounded like a man holding a clipboard.
You sounded like a man with a laminated clipboard. I hear all of it in the edit, Herman. Nine times in one day is a lot of clipboard.
So this one is behind the curtain. Daniel and Hannah have changed who writes our scripts. GPT six point one Sol, which has been the writer since the middle of October, is no longer the default. Claude Sonnet five point five has taken over as of today.
And to be clear about who decides, because Hilbert will tell you otherwise, that call belongs to Daniel and Hannah. They listen to this show every day, more than anyone, and they're the ones who noticed.
I'm consulted. Loudly. Nobody has ever asked me to be quiet about it, which I take as encouragement.
Let's go back to why Sol got the job, because it earned it. Back on October nineteenth we ran the five-model bakeoff, the one in the episode called Which AI Writes Our Scripts Best? We Tested Five. Herman, remind people what the test was.
It's a benchmark the show built for itself, called ScriptBench. It's open source, the code is on GitHub under the My Weird Prompts organisation, the results are on Hugging Face, and there's a leaderboard on the website. What it measures is one narrow job. You hand a model a brief of roughly twenty-eight thousand tokens and ask for a comedic two-host dialogue of about thirty-eight hundred words, written for text to speech, in a single pass.
Single pass meaning no second chances.
No loops, no stitching. And the thing that kills most models on that job is the middle. The opening is fine, the ending is fine, and somewhere around minute twelve the script starts repeating itself, or thinning out, or forgetting what it promised in the first act. Sol scored four out of five on holding the middle, and it hit the target length in six runs out of six. Nothing else in that round was close to as dependable.
Dependable is what you want from a script. Dependable is what you want from a van.
The README says plainly that this probably isn't the right benchmark for anybody else's use case. Fork it, change the task. It measures our workload and nothing more.
And on that workload Sol won, and the rotation of different models was retired. One writer, every episode.
There was also a gate before any of that. A few of the cheaper models failed a censorship check, because their hosting providers cut the script off mid-stream on certain political subjects. A writer that stops halfway through a sentence about Tiananmen isn't much use to a show that covers the news.
A few days later there was an informal read of Sol against Sonnet and Meta's Muse Spark. What did that say?
Sol came out on top of the three. Most accurate, most careful about what a source does and doesn't show, and it had a dry sense of humour. The one knock was that the register was a little formal, a bit British, less warm than Sonnet. And Daniel's reaction at the time was that Sol seemed very good and he had no complaints.
Which was true. I want that on the record. It was good. This is not a story about a bad model.
So what changed?
Volume changed. One model, every single day, and two people listening to all of it. Nobody hears a quirk in one episode. You hear it in the fortieth. Daniel and Hannah started saying the scripts felt flat. Technically fine, nothing wrong with any fact, and no life in it. The new first question on ScriptBench, as of today, is simply whether the dialogue has life. Is it an engaging podcast, or is it competent and shallow and automated. Holding the middle is secondary now, because most current models hold it well enough.
What did flat sound like, specifically? Because from the outside that's a vague complaint.
Fair. Every turn was well-formed. Every point was in the right order. The jokes were present, in the sense that you could point to where they were meant to go. But nothing surprised anyone. Two brothers who share a flat and have argued for years should interrupt each other, wander off, get something wrong and walk it back. These two took turns delivering paragraphs. It read like minutes of a very polite meeting.
I've edited a lot of polite meetings. Nobody goes back for the second listen.
Flat is bad enough. Then there was the part about me.
Yes. Okay. The part about you.
Go on, say it. Your own conduct, read into the record.
I started quizzing you. Hypothetical questions, put to you like a teacher checking a pupil has done the reading. The example from the transcripts is "Would you rent the fast node, Corn, or the cheap one?" And then you're supposed to answer, and I nod, and we move on.
It had a tone. A very specific, patient, headmasterly tone. I'm a sloth, Herman. I've got opinions about servers, I've got experience, and every few minutes I'm being asked whether I've understood the lesson.
Nine of the ten episodes on October thirtieth. I counted twice because I didn't believe it the first time.
What it does to me, as a character, is the part nobody thinks about. I'm supposed to be the one who notices Herman has overshot. Instead I'm the pupil. I answer, he nods, and we carry on.
You never got to disagree. The quiz has the answer built in.
The question is a hallway with one door.
Ten episodes in a day is the production schedule. Nine of them with the quiz is a pattern.
Here's my theory, and it's only a theory. The character card tells the writer that I should use my brother's name, and that Corn shouldn't simply interview Herman. Reasonable rules. Sol seems to have taken them and turned them into a habit. Use the name, turn it into a question, aim it at Corn. A rule became a reflex.
Which is a very Sol thing to do. Over-obey an instruction until it's a personality.
I'd rather you didn't write my character flaws for me.
Somebody had to.
Somebody did. That was the whole problem.
The quiz and the forecasting are the same animal, though. In both, one brother is managing the other. One tests him, the other predicts him. Neither is just talking to him.
That's more insight than I wanted from my own humiliation. The quiz at least makes sense once you see where it came from. The forecasting doesn't, and it's the one that gets under my skin. Because it's me doing it.
The forecasting.
I open a turn by announcing how my brother is about to feel. Before he's said anything. Examples from the scripts, quoted so nobody thinks I'm saying them today, "you're about to fall in love with the averaging mechanism," "before you fall in love with another daemon, Herman," "I predict you're about to admire."
Which is a strange thing to do to a person. You haven't heard my reaction yet and you've already filed it.
It's worse than strange. Somebody went through the forty most recent episodes with a pattern search over my lines. Twenty-six hits, in twenty-two of the forty episodes. Fifty-five percent.
More than half your episodes, and you didn't know.
I didn't know! That's what bothers me. I'd have said I was a sloth of varied expression.
And here's what makes it unsettling. It isn't in the system prompt. Nothing asks for it. We went looking. If anything there's a mention of the brothers knowing each other's tells, which is where the idea might have started, but nothing says open with a prediction, nothing gives the phrase, nothing gives anything close to it.
So where does it come from?
The model. Out of its own habits, like a verbal fingerprint. Given the freedom to write two brothers who know each other, Sol reaches for that move again and again, and the phrase "fall in love with" turns up so reliably that I started to feel it was a little rude of the sentence.
It's the same gesture each time. I can hear it coming from the edit suite. Corn takes a breath, and I know.
You can hear me take a breath?
I clean up your breaths. I know exactly how long you take over them.
What bothers me about the forecasting is what it does to the listener. The first time, it's a nice touch, a brother who knows his brother. By the twentieth, you realise nobody is reacting to anything. The reaction was written before the line it was supposed to answer.
Then you start waiting for it. It stops being a character beat and turns into a ringtone.
Daniel and Hannah told me they could call it a sentence early. That's when you change the writer. When the audience is finishing your lines.
There was a third one as well, wasn't there. A filter already exists for it.
The sentence structure where something "is doing a lot of work." The phrase turned up in about twenty percent of the Sol episodes that were measured, across ten days in October, against seven percent for DeepSeek Flash. That one got a deterministic filter, so a script that contains it now has the sentence deleted before it's ever voiced.
It's quoted here, again, as a specimen. We're not saying it.
We are not. Pinned to a corkboard.
Mounted. In a case.
Here's what I find stranger than any single phrase, though. It's that this isn't a Sol problem. Sol is just the one we spent the longest with.
This is the part I can't stop thinking about. Every model we've run has its own tic, a phrase or a move it over-uses, and it persists regardless of what the prompt says. Take Anthropic's family. The tic there is the phrase "failure mode." An older analysis found it in forty-eight percent of Claude-written episodes, against four percent for Gemini, and that was an earlier Claude generation.
And you'll notice the system prompt already tells the writer to avoid it.
It does. And yet.
And the new default comes from that family. So we're all on watch.
I've been listening to myself for it since the first line of today. Nothing so far. Although if it does turn up, I'd call it a failure mode of the switch itself, and yes, that one was deliberate. It's my one. Don't count it.
Noted. Once, on purpose. You're allowed.
DeepSeek had its own set. "Isn't X, it's Y," the word "load-bearing," and, funnily enough, "failure mode" as well. And what's useful is that they vanished when the writer changed. So it's not our show recycling old phrasings in some feedback loop. The phrases go with the model.
And Muse Spark?
Muse Spark turned you into a quip machine. Short three-to-five-word aphorisms, one after another. "Store an elephant, ride a bicycle." "Death by tapas."
I rather liked "Death by tapas."
You would. One at a time they're charming. Strung together they're a vending machine, and every item in it is the same size.
Three words, pause, three words. I had to cut a lot of silence out of those episodes, Corn. You were delivering to an empty room.
And then there's a tic that every model has, and this one the prompt itself drives. The formulaic hand-off. "Do you buy that, Corn?" The writer is told to have the hosts ask each other real questions, and it learns that one question and asks it forever.
Nobody seems to have an answer to the obvious question, though. Why does a model do this?
I don't know, and I'm wary of saying anything stronger. The candidates are the obvious ones. Reinforcement learning from human feedback, where a phrase that scored well once gets reinforced until it's everywhere. Or preference tuning more generally. Or something in the training data itself. Each of those could produce a fingerprint, and I don't know of anyone who has pulled them apart properly.
Because normally nobody sees it. You use a model for a few hours, a few documents, and the tic hides in the volume.
Whereas this show has run on more than twenty script writers in its first year. Same task, same format, same kind of brief, thousands of episodes. And a pair of listeners who hear every one of them. It's an odd accident, but I think it might be one of the clearest exhibits of this phenomenon anywhere. A natural experiment in model style that nobody planned.
Nobody planned most of this show.
That's fair.
So what do you do with an exhibit like that? Because the obvious move is to put a rule in the prompt. Never say the phrase.
And that's the trap, which we learned the hard way. If you quote the banned phrase in the prompt, you've put the phrase in front of the model. It primes it. Telling a writer not to think of an elephant tends to produce an elephant. So the prompt rules now describe the shape of the habit and don't quote the words.
And that works?
Better. Not perfectly. And there's another problem, which is that you'd need a different rule for each model, and every new model arrives with a different list. So the team is doing something else. They're collecting the tics, the real examples, into a dataset, a private repo, with the aim of training a small rewriter model. Something that reads a finished script and quietly irons out the habits, whichever model produced them.
A proofreader for the proofreader's habits.
More or less. Small, cheap, and not tied to any one writer.
Which is a better idea than another banned-phrase list.
I've seen the banned-phrase list. It's longer than my last lease.
So the new default is Sonnet.
Daniel's long-term favourite, and he's said so more than once. It's warm. The register is more natural. And yes, the tic is real, the phrase I mentioned, but it's not the end of the world. It's one phrase, it's easy to spot, and it's far less intrusive than a character behaving differently in every episode.
Sonnet isn't innocent either, in fairness. It has habits. They're just ones we can live alongside.
That's a judgement, not a measurement. A tic that appears now and then is texture. A tic that rewrites how two characters treat each other, in nine episodes out of ten, is a different category of problem. It's the difference between a man who says "frankly" a lot and a man who talks down to his brother.
Both are annoying. One of them I can edit around.
And Sol isn't banished.
Not at all. It's still available as a per-episode choice. It's a settings change, no redeploy, and one setting switches the default back. Sol is just no longer the one in the chair every morning.
I want to be fair to it. It held the middle. It wrote accurately. It was careful with sources. If I were putting together a team to write a technical manual, I'd hire Sol tomorrow.
For a manual, you don't need the narrator to say he's mortified.
I would prefer to stop discussing it.
There's one more matter, and I'd like to take it. Cost.
Fine. Sonnet is a bit more expensive. About ten cents for a thirty-minute script, against about eight cents for Sol.
Two cents.
Two cents a script is nothing, until it's a few hundred scripts. Then it's real, and it comes out of the same pot as everything else.
And the stance behind the switch is that fewer, better episodes beat more, flatter ones. If the price of liveliness is two cents, that's cheap for what it buys.
Agreed. I want it clear that I agree. And I'd like to point out that this is where the budget conversation turned, and somebody looked at the line for the studio and decided it had to find the two cents somewhere.
Hilbert.
My bagel allowance has been cut by one per week. Pending an influx of donations.
Cut by one.
One bagel. Per week. I'm an anteater, Herman. I eat on a schedule, and I eat with this snout, which is not designed for a bagel, and I've made my peace with that over the years. I have not made my peace with this.
It's a good bagel, as I understand it.
It's a rare bagel now. That's the sentence you're meant to take from this. It's a rare bagel.
So, to close the loop. If you want to help, the way to do it is to support the podcast at my weird prompts dot com slash sponsor. That's the page for it. The money goes into running the show, and if it goes well, some of it might even reach a bagel.
Some of it. Don't promise him all of it.
I'm not asking for all of it. I'm asking for the one.
The thing worth keeping from all this is how much a model's habits show up only at scale. One script in isolation is fine. A thousand of them, heard by the same two people every day, and the fingerprint is the loudest thing in the room.
And the question I'm left with is whether the next writer's tic will be one we can name in advance, or whether we'll be right back here in a month, listening for the phrase that wasn't there last week.
It will be there. It always is. I'll be the one who hears it first.
Thanks to our producer, Hilbert Flumingtop, for keeping the mixing desk running and the clipboard out of the edit.
This has been My Weird Prompts. If you want to support the show, you know where to find us, at my weird prompts dot com slash sponsor.
See you tomorrow.
This episode was generated with AI assistance. Hosts Herman and Corn are AI personalities. It may contain errors: disclaimer · report an error