#4971: Fine-Tune vs Prompt: Taming Hallucinated Deadlines

Big models add fake deadlines. Small models miss tasks. Which wrongness can you build a system around?

Featuring
Listen
0:00
0:00
Episode Details
Episode ID
MWP-5153
Published
Duration
34:24
Audio
Direct link
Pipeline
V5.2
TTS Engine
chatterbox-regular
Script Writing Agent
deepseek-v4-pro

AI-Generated Content: This podcast is created using AI personas. Please verify any important information independently.

The problem of turning a rambling voice dictation into structured JSON looks simple until you actually try it. A large prompted model will produce valid JSON most of the time — but it will also quietly insert a deadline you never stated, because "end of month" triggers the model's pattern-completion instinct to fill a date field. That's not a parsing error; it's the model being helpful in exactly the wrong way.

Fine-tuning shifts the calculus, but not in the way most people expect. A fine-tuned model learns your specific schema at the weight level, making null a legitimate answer for fields where no value was implied. It reduces the hallucinated deadline problem — but only the kind where the model guessed yes because most training examples had a populated field. It won't stop the model from misinterpreting "soon" as a specific date, because that's a judgment call, not a formatting issue.

The real cost of fine-tuning for personal extraction tasks isn't compute — it's attention. A minimum viable dataset runs three to five hundred examples, with the sweet spot around eight hundred to twelve hundred. At five minutes per example, that's a hundred hours of Daniel labeling his own transcripts with consistent priority judgments. The model doesn't just learn what he wants; it learns that he knows what he wants. Inconsistent labels collapse the model's confidence back into guessing. The tradeoff ultimately comes down to automation tolerance: if there's a human review step, the big model's creative additions are catchable. If there isn't, the fine-tuned model's predictable narrowness becomes a safety feature.

Downloads

Episode Audio

Download the full episode as an MP3 file

Download MP3
Transcript (TXT)

Plain text transcript file

Transcript (PDF)

Formatted PDF with styling

#4971: Fine-Tune vs Prompt: Taming Hallucinated Deadlines

Corn
Daniel's been thinking about the same problem again, but this time on the text side. He's dictating a rambling brain-dump of a to-do list — tasks mixed with asides, corrections mid-sentence, commentary that isn't a task at all — and he wants clean structured JSON coming out the other side. Call the accountant before end of month, sort out the garage at some point, and the system has to know those are two objects with different priorities, one with a date and one without. The question is the one that keeps coming up: is this a fine-tune, a small purpose-trained model, or just system prompting on a large general model? And he doesn't want "it depends" — he wants the reasoning that decides it.
Herman
The line people keep missing is that this isn't one question, it's three stacked on top of each other. Format adherence, judgment quality, and failure predictability. They move in different directions when you change the approach, and picking one means accepting what happens to the other two.
Corn
So the format adherence part — that's the one where a big model with a good prompt and a schema mostly gets it right, until it quietly inserts a deadline nobody said.
Herman
And that's the failure mode that makes this whole thing interesting. A large model prompted for structured extraction will hallucinate fields. Not wildly — it won't invent a task about booking a flight to Mars — but it will add a priority level you didn't imply, or a deadline that felt plausible to the model given the phrasing. "Before the end of the month" becomes a specific date the model picked because it knows what "end of month" usually means in JSON schemas. That's not a parsing error, it's the model doing exactly what it was trained to do — complete patterns — and the pattern of a task object with a deadline field is stronger than the instruction to leave it null when uncertain.
Corn
The model is pattern-completing its way into wrongness.
Herman
Right. And the maddening thing is it'll get ninety-three percent of tasks perfect. You'll trust it. Then you'll miss the call with the accountant because the model decided "end of month" meant September thirtieth and you didn't catch it in review.
Corn
Which you won't review, because the whole point of automating this is to not review it.
Herman
So the format adherence question isn't really about whether the JSON is valid — structured output modes have mostly solved that. It's about whether the values inside the valid JSON are faithful to what was actually said. And that's judgment, not formatting.
Corn
So let's pull those apart properly. Format adherence, judgment, failure predictability. Where does fine-tuning actually move the needle, and where is it just expensive ceremony?
Herman
Fine-tuning moves format adherence more than people expect, but not in the way they think. A fine-tuned model doesn't just output valid JSON more often — the base models already do that with structured output modes. What fine-tuning does is make the model's internal representation of the schema match your specific schema. Every field, every enum value, every optional versus required distinction gets baked into the weights. The model stops having to "remember" the schema from the system prompt — it becomes the default shape of its output for this kind of input.
Corn
So the hallucinated deadline problem — does fine-tuning actually reduce that?
Herman
It reduces one specific kind of it. The kind where the model is uncertain about whether a field should be populated and guesses yes because most training examples had it populated. If you fine-tune on examples where the deadline field is explicitly null in cases where no deadline was stated, the model learns that null is a legitimate answer. The base model's instinct is to fill fields — that's what language models do, they predict tokens, and empty is harder to predict than something. Fine-tuning can teach it that empty is correct.
Corn
But it won't stop the model from misinterpreting "soon" as a date.
Herman
No, and that's the judgment part. Judgment is about understanding what counts as a task, what the priority actually is, whether "at some point" means low priority or just no deadline. That's much harder to fine-tune because the ground truth is subjective. Daniel's definition of "urgent" might not match what a rater on a labeling platform would mark. And if the training data doesn't capture his specific judgment, the fine-tuned model just learns someone else's judgment more consistently.
Corn
So fine-tuning for judgment requires Daniel to label the data himself.
Herman
Or someone who thinks like him. Which is usually just him. So now we're talking about a dataset of — what — five hundred to two thousand examples of rambling voice transcripts paired with the JSON he would have produced. That's a week of work just to generate the training data, and every example has to be his own judgment or the model learns the wrong thing.
Corn
And if he does that, what does he actually get? Better judgment, or just judgment that's wrong in a more consistent way?
Herman
Both, and that's the tradeoff. A fine-tuned model will be wrong in a more consistent way. If Daniel consistently treats anything with a date as high priority, the model will learn that pattern and apply it reliably. A prompted general model might apply that rule seventy percent of the time and randomly diverge the other thirty percent. The fine-tune gives you predictability — you know how it'll fail because you trained it on your own judgment and your own judgment has patterns.
Corn
Which brings us to the failure pattern question. Daniel's framing is that a big model fails by being quietly wrong in plausible ways, and a small fine-tuned model fails by being brittle when the input drifts. Is that right?
Herman
Mostly. The big model's failures are sins of commission — it adds things. Deadlines, priorities, sometimes whole tasks inferred from context. "You mentioned your wife's birthday in passing so I created a task to buy a gift." That's the model being helpful in exactly the way you don't want.
Corn
The small model's failures are sins of omission. It misses tasks entirely because the phrasing was slightly different from anything in training.
Herman
Or it splits one task into three because a pause in the dictation looked like a boundary and the training data had shorter utterances. The brittleness is real, but it's also predictable. You can test it. You can run a hundred examples through and find the exact phrasings that break it, then decide whether those phrasings actually occur in your dictation style.
Corn
So the question becomes: which kind of wrongness can you build a system around?
Herman
That's exactly the right question. And the answer depends on whether you're willing to add a review step. If Daniel is going to glance at the output before it goes into his task manager, the big model's hallucinated deadlines are catchable. He'll see "call accountant — due September thirtieth" and think wait, I didn't say that. But if this is fully automated — dictation to task manager with no human in the loop — the big model's creative additions are dangerous in a way the small model's omissions aren't. A missed task is annoying. A fabricated deadline is a missed commitment.
Corn
A missed task you might remember. A fabricated deadline you'll trust.
Herman
And act on. Or fail to act on and blame yourself. So the automation tolerance is the first thing to nail down. If there's a review step, the big model probably wins on cost and flexibility. If there's no review step, the fine-tuned model's predictable narrowness starts looking like a safety feature.
Corn
Let's talk about the data question. Daniel asked what the labelled data actually looks like when the ground truth is his own subjective sense of what counts as a task.
Herman
This is where most fine-tuning projects for personal use fall apart. The data isn't just input-output pairs. It's input paired with output that reflects one person's judgment, consistently applied, across enough examples that the model can find the pattern. For Daniel's to-do list case, each training example is a transcript on one side and a JSON array on the other. The transcript is the easy part — you can generate those by talking, or by taking existing transcripts and deliberately making them messier. The JSON is the hard part because Daniel has to write it himself for at least a few hundred examples before the pattern is learnable.
Corn
And he has to be consistent.
Herman
Brutally consistent. If "sort out the garage" is medium priority in one example and low priority in another with similar context, the model learns that priority is random. The training data doesn't just teach the model what Daniel wants — it teaches the model that Daniel knows what he wants. If the labels are inconsistent, the model's confidence collapses and it starts guessing, which is exactly what the base model does anyway.
Corn
So the fine-tune doesn't just require effort — it requires a level of self-knowledge about your own categorization habits that most people don't have.
Herman
And won't discover until they're fifty examples in and realize they've been inconsistent about what "high priority" means. That's actually a useful exercise on its own, but it's not free.
Corn
How many examples are we talking about, practically?
Herman
For this kind of extraction task, the minimum viable fine-tune is probably around three hundred to five hundred examples. Below that, the model doesn't have enough signal to override its base instincts about when to fill a field. Above two thousand, you're usually into diminishing returns unless the task is extremely subtle. The sweet spot for something like Daniel's to-do list is probably eight hundred to twelve hundred examples, assuming he's consistent in his labeling.
Corn
And what does that cost?
Herman
If he's using a smaller model — say a seven or eight billion parameter model — the fine-tune itself might cost fifty to two hundred dollars in compute, depending on the provider and how many epochs he runs. The real cost is the labeling time. At five minutes per example, twelve hundred examples is a hundred hours. That's the part people don't budget for.
Corn
So the fine-tune is cheap in compute and expensive in attention.
Herman
Which is exactly inverted from what most people assume. They think the compute is the barrier and the data is just something you gather. But for personal, subjective extraction tasks, the data is the whole thing. The model is just a way to persist your judgment so you don't have to apply it manually every time.
Corn
Let's pull on the thread Daniel raised about text versus vision. He said text transformation feels like the case where the general model has run away with it most completely. Is that right?
Herman
It is, and the reason is structural. Text-to-text transformation is what these models were built to do. The pretraining objective — predict the next token — is already a text transformation task. Structured extraction is just next-token prediction with a constraint on the output format. The model has seen millions of examples of people describing things and then formatting them. It's not learning a new skill — it's applying a skill it already has to a slightly narrower domain.
Corn
Whereas vision-to-text is crossing modalities.
Herman
Right. A vision language model has to map pixels to concepts to text, and the structured extraction is happening at the end of a longer pipeline with more places for things to go wrong. The text case is closer to the model's native operating mode, which means the base model starts from a much higher baseline. A good prompt and a schema on a large model might get you to ninety percent accuracy on text extraction. The same approach on vision might get you to seventy percent.
Corn
So fine-tuning has more room to improve things in the vision case.
Herman
Much more. In text, fine-tuning is squeezing out the last five to eight percent of errors. In vision, it might be squeezing out twenty percent. The cost-benefit calculation is completely different.
Corn
Which means for text, the bar for fine-tuning being worth it is higher. You're paying the same labeling cost for a smaller improvement.
Herman
And that's why "it depends" is the answer everyone gives but nobody finds useful. The real reasoning is: how much does each percentage point of accuracy cost you in the failure pattern you actually care about? If the big model's hallucinations would cause you to miss a tax deadline, and the fine-tuned model's omissions would cause you to forget to clean the garage, those are not symmetrical. The cost of a wrong answer depends on what you're extracting.
Corn
So the decision isn't about the model — it's about the stakes of the specific extraction task.
Herman
And the volume. If Daniel is dictating three to-do lists a week, the big model's occasional hallucination is manageable. He'll catch it. If he's dictating fifty a day and they're flowing directly into a task management system that sends him notifications, the hallucinations compound and he stops trusting the system entirely.
Corn
There's a trust decay problem here that nobody talks about. One hallucinated deadline and you start reviewing everything, at which point the automation saved you nothing.
Herman
The system's utility is the accuracy rate multiplied by your willingness to not review the output. If accuracy drops below some threshold — and that threshold is personal — your review behavior changes and the whole thing collapses. Fine-tuning buys you a higher accuracy floor, which buys you more months of not reviewing before the first error erodes your trust.
Corn
What about the small purpose-trained model option? Daniel mentioned that as a third path — not fine-tuning a general model, but training something tiny and specific.
Herman
For text extraction, that's almost never the right call in 2026. A small model trained from scratch on extraction tasks will be worse than a fine-tuned general model at everything except speed and cost per token. The general model brings world knowledge that matters for this task — knowing that "the accountant" implies a phone call, knowing that "end of month" means a date constraint, knowing that "sort out the garage" is a physical task with no inherent deadline. A small model trained only on extraction examples loses all of that. You'd have to train it on a massive corpus of general text first, at which point you've just built a worse version of the general model.
Corn
So the real choice is between prompt engineering on a large model and fine-tuning a medium model.
Herman
With one exception. If the extraction task is extremely narrow and high-volume — say, extracting exactly three fields from a million customer service transcripts where the domain is fixed and the vocabulary is limited — a small purpose-built model can work. But Daniel's to-do list is the opposite of that. The domain is his entire life. The vocabulary is unbounded. The task definitions are subjective. That's exactly the kind of problem where world knowledge matters.
Corn
Let's talk about the system prompt approach, because I think there's more mileage there than the fine-tuning advocates admit. What's the state of the art for prompting structured extraction?
Herman
Structured output modes have gotten good. You give the model a JSON schema — either as a Pydantic model, a Zod schema, or just raw JSON Schema — and the model is constrained to produce tokens that conform to it. Not prompted to conform, constrained. The sampling is modified so that invalid tokens are impossible. That eliminates the malformed JSON problem entirely.
Corn
But it doesn't eliminate the wrong values inside the valid JSON.
Herman
No, and that's the limit. Constrained generation guarantees the shape is correct. It does nothing about whether the deadline field contains a date that was actually mentioned. That's still a judgment problem, and judgment lives in the model's weights, not in the sampling constraints.
Corn
So the system prompt is doing two things: defining the schema, and giving the model instructions about how to make judgment calls. The schema part is solved. The judgment part is where the prompt has to do real work.
Herman
And the prompt can do more work than people give it credit for. A well-written prompt for this kind of extraction should include not just the schema but examples of edge cases and the reasoning behind them. "If the speaker says 'at some point' or 'eventually,' do not assign a deadline. If the speaker says 'urgent' or 'as soon as possible,' set priority to high. If a statement is a comment about a task rather than a task itself, do not extract it." That kind of explicit decision boundary in the prompt can catch a lot of the failures that people assume require fine-tuning.
Corn
But the prompt gets long, and long prompts have their own failure pattern. The model starts attending to the wrong parts.
Herman
Or the instruction for one edge case bleeds into another. "Do not assign a deadline for 'at some point'" might make the model hesitant to assign deadlines for anything vague, including "next week" which is vague but definitely implies a deadline. The prompt becomes a set of competing heuristics, and the model has to adjudicate between them on every example.
Corn
Which is exactly what fine-tuning solves. The heuristics are baked into the weights, and the model doesn't have to hold them in its context window while also processing the input.
Herman
That's the core insight. Prompting is asking the model to apply rules it's reading for the first time. Fine-tuning is making the rules part of how the model thinks. The difference shows up most in consistency across long inputs. If Daniel dictates for three minutes and mentions fifteen things, the prompted model might forget the "don't extract commentary" rule by the end because the context window is full of transcription and the rule was at the beginning. The fine-tuned model doesn't forget because the rule isn't in the context — it's in the model.
Corn
So the line between prompting and fine-tuning isn't about task difficulty in the abstract. It's about whether the rules are stable enough and applied frequently enough that they're worth memorizing.
Herman
And whether the cost of forgetting a rule mid-input is high enough to justify the labeling effort. For Daniel's to-do list, if he tends to dictate long rambling sessions, the context-window attention problem is real. The model might handle the first two minutes perfectly and then start extracting commentary as tasks in the third minute because the system prompt has been pushed out of the effective attention window.
Corn
What about the hybrid approach? Fine-tune on a few hundred examples, but still include a short system prompt with the schema and the most critical rules?
Herman
That's usually the sweet spot. The fine-tune handles the consistent judgment patterns — what counts as high priority, when to leave deadline null, how to segment tasks. The system prompt handles the schema definition and any rules that might change. "This week, tag anything about the garage as high priority because we're renovating." That kind of temporary override belongs in the prompt, not in the weights.
Corn
Because you don't want to re-fine-tune every time your garage situation changes.
Herman
Right. The fine-tune captures your stable judgment. The prompt captures your current context. Separating those is the difference between a system that works for years and one that works for two weeks.
Corn
Let's go back to the failure pattern comparison Daniel asked about. He said a big model hallucinates deadlines or splits tasks wrong, while a small fine-tuned model is brittle to input drift. What does "input drift" actually look like in practice?
Herman
The most common failure is when the dictation style changes. If Daniel's training data is all short, focused dictations — "call accountant, deadline end of month" — and then one day he dictates while driving and the input is full of traffic noises, false starts, and tangents about the song on the radio, the fine-tuned model sees a distribution it wasn't trained on. It might start extracting the song lyrics as tasks. It might fail to segment anything because the signal-to-noise ratio is different from training.
Corn
The base model handles that better because it's seen everything.
Herman
The base model has seen drunk driving dictations, crying dictations, dictations from people who forgot what they were saying mid-sentence. Its training distribution is the entire internet. A fine-tuned model's training distribution is whatever Daniel chose to include, which is probably his best, cleanest dictations because those are the ones he bothered to label.
Corn
So the fine-tune trains on ideal conditions and deploys in messy ones.
Herman
And that's a solvable problem, but you have to know to solve it. You deliberately include messy examples in the training data. Dictations with background noise, with interruptions, with tangents. Label them correctly despite the mess. That teaches the model that mess is normal, not a distribution shift.
Corn
Which adds more labeling hours to an already expensive process.
Herman
It does. But it's the difference between a model that works in a demo and a model that works on a Tuesday morning when you're running late and dictating while making coffee.
Corn
I want to circle back to something Daniel said about the vision comparison. He suggested text transformation is where the general model has run away with it most completely. I think that's right, but for a reason he didn't mention. Text transformation is the only case where the input and output are in the same modality, which means the model can use its own internal representations directly. It's not translating — it's reshaping.
Herman
That's a sharper way to put it. In vision extraction, the model has to translate pixels to concepts to text, and each translation step is a place where information can be lost or distorted. In text extraction, the concepts are already in the model's native format. The only translation is from unstructured natural language to structured natural language, and the model's internal representations are already good at that because that's what the entire pretraining process was about.
Corn
The baseline is higher, the improvement from fine-tuning is smaller, and the decision to fine-tune has to be justified on narrower grounds.
Herman
Those narrower grounds are usually about consistency and trust, not about capability. A prompted large model can do this task. It can do it well. The question is whether it can do it well enough, consistently enough, for long enough that Daniel doesn't feel the need to review every output. If the answer is no, fine-tuning buys him consistency. If the answer is yes, fine-tuning is a solution in search of a problem.
Corn
There's one more dimension Daniel raised that we haven't touched. He asked whether format adherence and judgment are actually the same thing, or whether fine-tuning buys one but not the other.
Herman
They're different, and fine-tuning buys both but through different mechanisms. Format adherence improves because the model internalizes the schema — it knows that the output shape is always an array of objects with these specific fields, and it stops having to consult the system prompt to remember that deadline is optional. Judgment improves because the model internalizes Daniel's specific decision patterns — what he considers a task versus commentary, what he considers high priority, when he assigns a deadline.
Corn
But the judgment improvement is only as good as the consistency of the training data.
Herman
That's the catch. If Daniel's judgment is inconsistent — if he sometimes extracts a task from "I should probably think about the garage" and sometimes doesn't — the fine-tune won't improve judgment at all. It'll just make the inconsistency faster. The model will learn that the correct answer is a probability distribution over possible extractions, and it'll sample from that distribution, which is exactly what the base model does.
Corn
Fine-tuning amplifies your judgment, but it also amplifies your inconsistency.
Herman
It's a mirror. If you know what you want and you can express it consistently across five hundred examples, fine-tuning will make a model that wants the same thing. If you don't, fine-tuning will make a model that's as confused as you are, just faster.
Corn
Which brings us to the uncomfortable conclusion that for a lot of personal extraction tasks, the bottleneck isn't the model — it's the human's ability to define what they want consistently.
Herman
That's the thing the "just fine-tune it" advice always misses. Fine-tuning is not a way around the hard work of defining your task. It's a way of paying off the hard work after you've already done it.
Corn
Where does Daniel actually land? He's dictating to-do lists, he wants structured JSON, he doesn't want to review the output. The volume is probably moderate — a few a day, not hundreds. The stakes are medium — missing a task is annoying but not catastrophic.
Herman
I'd start with the big model and a really good system prompt. Not the default "extract tasks as JSON" prompt, but something with explicit decision boundaries, edge case examples, and the reasoning behind each rule. Run that for two weeks. Collect every output that's wrong. Categorize the errors. If the errors are mostly the same kind of thing — consistently hallucinating deadlines, consistently missing tasks that are phrased a certain way — that's a signal that fine-tuning could help, because the errors are systematic.
Corn
If the errors are random?
Herman
If they're random, fine-tuning won't help. Random errors mean the model is fundamentally uncertain about the task, and the fix is either a better prompt or accepting that some error rate is inevitable and building a review step.
Corn
The systematic-versus-random error distinction is the diagnostic that nobody runs before jumping to fine-tuning.
Herman
Because running it requires two weeks of actually using the system and paying attention to the failures. It's much more satisfying to just fine-tune something and feel like you've solved the problem.
Corn
Even if you've just made the same errors faster and more consistent.
Herman
Which, to be fair, is sometimes exactly what you want. Consistent errors are build-around-able. If the model always assigns a deadline to "next week" as "next Monday" and Daniel actually means "sometime in the next seven days," he can write a post-processing rule that strips those deadlines. You can't post-process around random hallucinations.
Corn
The decision tree is: try prompting first, categorize the errors, if they're systematic and costly enough to justify a hundred hours of labeling, fine-tune. If they're random, fix the prompt or add a review step.
Herman
If the volume is high enough that even a five percent error rate means twenty corrections a day, fine-tune even if the errors are random-ish, because the labeling cost is amortized over enough corrections avoided.
Corn
The volume threshold is the thing most people don't calculate. A hundred hours of labeling at whatever Daniel's consulting rate is versus ten minutes a day of correcting outputs for the next two years. One of those numbers is usually much bigger than the other.
Herman
It's almost always the labeling cost that wins for personal tools. A hundred hours is two and a half weeks of full-time work. Ten minutes a day of corrections is sixty hours over a year. If Daniel uses this system for more than two years — and he probably will, because to-do list dictation is not a thing you stop doing — the prompting-plus-correction approach is cheaper in time even before you account for the fact that labeling is miserable work.
Corn
Unless the corrections erode trust to the point where he stops using the system entirely.
Herman
Which is the intangible that makes this a judgment call rather than a spreadsheet. Trust decay is real, it's personal, and it doesn't show up in a cost-benefit analysis until the system is already abandoned.

Hilbert: You're both talking about this like the errors are the problem. The errors are fine. The problem is that nobody counts how many tasks they actually said.
Corn
Go on.

Hilbert: I spent eighteen months doing basically this. Not to-do lists. Insurance claims. People would call in, describe what happened to their car, and the system was supposed to pull out the structured claim — date, location, parties involved, damage description. We had a big model doing it with a prompt, worked great in testing. Then we ran it on real calls and the accuracy dropped to about sixty percent.
Herman
What was breaking?

Hilbert: People don't describe car accidents in the order the form expects. They start with how they feel, then mention the other driver was rude, then remember the intersection, then circle back to the damage. The model was extracting the first thing that sounded like each field and ignoring the correction that came forty seconds later. "He hit me on the driver's side — actually no, passenger side" — the model took driver's side and moved on.
Corn
The failure wasn't hallucination, it was not tracking revisions.

Hilbert: The model treated the transcript like a document where the first mention is the truth. Real speech is full of self-corrections, and the model wasn't listening for them. We ended up having to add a whole preprocessing step that identified when someone corrected themselves and flagged the original statement as superseded. That got us to about eighty-five percent. Never got higher than that without a human review.
Herman
Did you try fine-tuning?

Hilbert: We tried. Made it worse. The fine-tuned model got better at the format and worse at the corrections, because the training data was all clean final versions. Nobody had labeled the messy intermediate states. So the model learned that claims are well-structured and self-consistent, which real claims are not.
Corn
The training data taught the model that the world is cleaner than it is.

Hilbert: Same problem you were describing with Daniel's dictations. If he only labels his best dictations, the model learns that dictations are clean. Then his real dictations — the ones with "actually, scratch that" and "wait, I forgot to mention" — those break it.
Herman
How many examples did you label?

Hilbert: About six hundred. Took three weeks. The project lead was convinced fine-tuning would solve everything. It solved the JSON formatting, which was never the problem. The problem was that people talk in loops and the model wanted a straight line.
Corn
What happened to the project?

Hilbert: It's still running. They kept the prompted model with the correction-detection preprocessing. Added a review step for claims above a certain dollar amount. Everyone adjusted their expectations downward. The accuracy number stopped being the thing anyone cared about — what mattered was whether the errors that got through were cheap or expensive. A wrong date is expensive. A missing description of the other driver's attitude is not.
Herman
That's the triage approach. Don't try to eliminate errors — just make sure the errors that survive are the ones you can afford.

Hilbert: It's less satisfying than building a perfect model. But it ships.
Corn
Did you keep any of the fine-tuned model?

Hilbert: No. We kept the labeling guidelines, though. Those turned out to be more valuable than the model. Once you've written down exactly what counts as a valid claim and what doesn't, you've got something you can hand to a new person or feed into a better model later. The model was disposable. The definitions weren't.
Herman
That's a useful way to think about the labeling investment. Even if the fine-tune doesn't work out, the act of defining your task precisely enough to label examples is valuable on its own.

Hilbert: It's the only part of the project I'd do again. The rest was just... I spent three weeks arguing with a claims adjuster about whether "I was shook up" counts as an injury. It doesn't. But the model thought it did because "shook up" sounds medical. You can't fix that with more examples. You fix it with a rule.
Corn
A rule in the prompt, or a rule in post-processing?

Hilbert: Post-processing. The model extracts everything that might be relevant. Then a much dumber system — basically a list of patterns — strips out the things that aren't actually claims. The dumb system is easier to trust because you can read the rules.
Herman
Layering a reliable dumb system on top of an unreliable smart system. That's a pattern that shows up everywhere once you start looking.

Hilbert: It's not elegant. But the car got fixed either way.
Corn
I keep thinking about what you said about the definitions outlasting the model. Daniel's real asset here isn't the fine-tuned weights — it's the document that says "this is what I mean by a task, this is what I mean by high priority, these are the phrasings that sound like tasks but aren't." Whether that document lives in a system prompt, a fine-tune, or a post-processing layer is a deployment decision. The document is the thing.
Herman
That document is what makes the whole system maintainable. Six months from now, when a new model comes out that's better at extraction, Daniel can take his definitions and write a new prompt for the new model in an afternoon. If he'd baked everything into a fine-tune, he's starting over.
Corn
The fine-tune is a commitment to a specific model at a specific point in time.
Herman
Which is fine if the task is stable and the model is good enough to last. But for personal extraction tasks where the models are improving every few months, locking yourself to one model's weights is paying a switching cost you're going to pay eventually anyway.
Corn
The final answer to Daniel's question — the reasoning that decides it, not "it depends" — is something like: start with the big model and a prompt that encodes your definitions. Run it until you have a catalog of errors. If the errors are systematic and the volume justifies the labeling cost, and you're confident your definitions are stable enough to be worth memorizing, fine-tune. Otherwise, invest in better definitions and a review step.
Herman
If you do fine-tune, keep the definitions document. The model is temporary. Knowing what you want is permanent.
Corn
Thanks to our producer Hilbert Flumingtop.
Herman
This has been My Weird Prompts.
Corn
If you want to send us your own weird prompt, email us at show at my weird prompts dot com.
Herman
We'll be back soon.

This episode was generated with AI assistance. Hosts Herman and Corn are AI personalities.