A corrected answer only helps if the next pass actually uses the correction. That's the part of Daniel's prompt I was turning over. We're talking about gold and silver training data today, and the point where a small amount of human checking is supposed to make a much larger machine-generated batch trustworthy.
I've done that staged process a few times. I let an LLM make a first pass, review some of it myself, then use those corrections for a bigger pass. I want to understand what that process is called, how much verified material it needs, and what I should keep separate for evaluation.
Your correction can change several different things. It can repair a label, clarify the instructions, or reveal that you're generating the wrong kinds of examples altogether. Those lead to different next passes.
You fix one answer and discover you've purchased a taxonomy.
Sometimes that's the valuable purchase. If a reviewer keeps changing the same kind of answer, you need the rule behind those changes, rather than a longer collection of corrected sentences.
That already sounds more useful than just adding the corrected examples to the prompt and hoping.
Hoping scales beautifully. Checking has staffing requirements.
Gold usually means human-annotated or human-verified reference data. Silver means automatically annotated data, whether an LLM produced it or an older algorithm did. The distinction concerns where the labels came from and what confidence you place in them. Both can arrive in exactly the same file format.
A spreadsheet cannot assay itself.
Human-written also needs a qualification. A person can write a careless answer. Calling something gold is a claim about the annotation process, ideally with clear instructions, review, and a way to resolve disagreements. Human contact alone doesn't confer correctness.
If I check a model-generated pair carefully, does that particular pair become gold?
It can. The starting material doesn't permanently determine its status. If you verify both the input and expected output against your task definition, that record can become part of a gold reference set. The unchecked neighbours remain silver.
I would keep the original label and your correction, too. Otherwise six weeks later you know somebody changed something, but not whether the model misunderstood the question or you changed your mind about the rules.
Corn, would you actually call the checked version ground truth?
For a postcode, perhaps. For whether a customer sounds annoyed, no. People can agree on a label because the manual tells them to, while still disagreeing about the customer. I'd call it the answer we're holding the system to.
That's why gold standard is often the more accurate phrase. Annotators disagree even on apparently straightforward tasks. The reference establishes a standard you can reproduce, rather than granting access to an uncontested truth.
Does that mean I need two people reviewing everything before I can use the word gold?
No universal staffing rule comes with the term. A specialist checking factual answers might be sufficient for a narrow project. A subjective classification task benefits more from independent annotation and adjudication. What matters is describing what you actually did.
Gold, with the receipt attached.
The medical use makes this clearer. A gold standard diagnostic test is the best available reference under reasonable conditions. You validate a new test against it. That reference can still miss cases, cost too much for routine use, or eventually be replaced.
You're about to defend a reference test against an accusation nobody made.
I spent years explaining why a negative result didn't erase the child in front of me. I'm allowed one sentence.
You've had three. They were relevant.
NLP borrowed that reference-standard language for human-verified annotations. The monetary metaphor is older, of course, but medicine gives us the useful operational meaning here: a benchmark against which something else is judged.
Silver sounds like somebody deliberately invented a cheaper alternative. Was it already used before LLMs?
Long before them. A useful early landmark in biomedical NLP is the CALBC project around two thousand ten. It combined the outputs of multiple named entity taggers. Those systems identified things such as genes, diseases, and chemicals in text, then their different annotation schemes were reconciled into a consensus.
A committee of taggers. No catering budget.
Hahn and colleagues called their proposal a configurable silver standard. Their attraction was speed, low cost, and huge amounts of annotation. They were also explicit that its applicability and validity still needed experimental testing.
Consensus between models feels reassuring. Unless they're all making the same mistake.
Agreement is cheap when everyone copied the same homework.
Shared training data, shared dictionaries, shared assumptions about entity boundaries... all can produce correlated errors. If every tagger excludes part of a disease name, a vote doesn't restore it. Consensus is evidence worth testing, not automatic promotion to gold.
Would you trust consensus more than one excellent model, Corn?
Only if the disagreements tell me something. Five near-identical models saying yes gives me less comfort than two different methods agreeing. I'd still want examples where they disagree with the person reviewing them. That's where the collection becomes informative.
Diversity among annotators matters as much as their number. It also explains why silver extends beyond synthetic text. You can take real documents, automatically attach labels, and have a silver corpus without generating a single new sentence.
Right. For my staged approach, what would you actually put in the project description? Something another developer would recognise.
Iterative synthetic data generation with human review. More specifically, iterative self-instruction with human label correction, if you're generating instruction and answer pairs. There's no single canonical name covering every version.
I'd put the plain description first. A data flywheel can conceal a surprising number of people pushing.
Self-Instruct is a particular framework for bootstrapping instruction-following data from a model's own generations. Generate instructions, inputs, and outputs, filter invalid or overly similar examples, then fine-tune. Repeating that loop with reviewed corrections makes it a close relative of what you're doing.
Does it still count if I haven't fine-tuned anything and I'm just changing the prompt between passes?
The broad workflow does. Calling it the exact Self-Instruct algorithm would be less precise. You're doing seed-based synthetic generation and iterative refinement. The corrections may steer a prompt, a retrieval bank of examples, or eventually model weights. Those mechanisms aren't interchangeable.
A model hasn't learned your correction permanently because it nodded at it.
The same workflow also overlaps with weak supervision when imperfect automated labels guide training, and with semi-supervised bootstrapping when verified and unverified material interact. Active learning is narrower: it helps choose which records a human should annotate next.
That's the distinction I was missing. Reviewing a subset doesn't automatically mean I've used active learning.
Randomly taking a slice is sampling. Selecting uncertain, disputed, or unusually informative records for review is closer to active learning. You can combine them. Keep a random sample to estimate ordinary quality, and a targeted sample to find expensive mistakes.
If you review only the most alarming records, your dataset looks terminally ill. If you review only easy ones, it gets discharged prematurely.
The staged process, with human review of a subset before scaling, is what Daniel is asking about. The selection policy determines what that review can tell you.
Ivan: Voronin, Voronin. Seven four, seven four. Two one, nine three. The cat will not come in. The cat will not come in until the silver bowl is empty. The gold dish is for the vet. Repeat: two one, nine three. The porch light is off. The tabby knows the ground truth. Kestrel.
Two one... nine three. Confirmed.
What on earth was that?
A useful review gate needs acceptance criteria. Valid structure, correct answers, coverage of the intended tasks, and enough variation that you're not buying duplicates with different names.
Suppose the first sample has a recurring error. Do I correct those records and carry on, or regenerate the batch?
I'd first ask whether the error has a boundary. A wrong spelling you can locate is different from an instruction that quietly produced the wrong interpretation everywhere.
Repair the local defect if you can identify it reliably. Revise the generation rule and rerun affected material if the defect is systematic. Then check fresh outputs, not only the examples you already repaired.
Automated judges can help triage, but their own mistakes need measuring. One closed-loop implementation reported roughly eighteen percent wrong judge calls. That's a result from that setup, not a universal error rate for LLM judging.
The inspector also needs inspecting.
Then a judge is more useful for finding patterns than certifying every row?
It can rank likely problems and reduce the review load. Whether it can accept individual records depends on the task and its measured reliability. A judge recognising fluent prose doesn't establish that a factual answer is supported by the source.
Especially when the generator and judge share a weakness. You can automate the disagreement away and retain the error.
The part I'm still trying to make concrete is the seed size. Am I looking for a few impeccable examples, or do I need a substantial dataset before this is worth doing?
A striking result comes from BARE. It starts with three seed examples, combines a base model with an instruction-tuned model, and generates a much larger dataset. Fine-tuning an eight-billion-parameter Llama model on a thousand generated coding samples matched state-of-the-art similarly sized models on LiveCodeBench.
Three examples. Finally, a reading assignment within my budget.
I knew you'd endorse the seed count before hearing the evaluation conditions.
I endorse the workload. You've mistaken that for peer review.
The benchmark qualification matters. That result shows a small seed can unlock capabilities the pretrained model already has. It doesn't establish that three examples define every task, or that a coding benchmark settles performance in your application.
Is the base model doing something an instruction model won't do?
In that pipeline, it supplies diversity. Instruction tuning can concentrate outputs into familiar patterns. Ask for many examples and you may get polished variations on a narrow template. The base model explores more possibilities, while the instruction-tuned model helps refine their quality.
The helpful model keeps serving the same meal on different plates.
Not all instruction models behave identically, but yes, that kind of narrowing is the concern. Raising sampling temperature doesn't necessarily restore missing kinds of examples. It can just make the same limited set less reliable.
I wouldn't want to conclude that base models are better generators full stop.
Nor would I. They're useful components in this particular division of labour. They need control and filtering. More varied output includes more unusable output.
I can generate unusable variety without loading anything.
The contrast in scale is enormous. OSS-Instruct used eighty thousand code snippets as source material. Humpback used over half a million text segments. Those aren't directly comparable units to three verified pairs, so treating them as rival minimum requirements would be misleading.
Source material gives you subjects to explore. A seed pair can show the required behaviour. Both get called seeds, and suddenly somebody thinks a label correction replaces a library.
Some of the apparent disagreement about seed size comes from that vocabulary. There's task demonstration, domain coverage, and raw material for generating new tasks. A pipeline may need very little of one and a great deal of another.
What about examples outside coding? Is this single-digit territory unusual?
It appears elsewhere. QAMELEON used five golden samples for synthetic question answering, and GeMQuAD generated usable multilingual question-answer data with one annotated target-language example. Again, pretrained capability is doing enormous work.
The one example didn't teach the model a language over lunch.
Meanwhile, a hundred examples appears in descriptions of OpenAI fine-tuning results. That's a different question: how much training data can begin to produce a useful adaptation. It shouldn't be presented as a universal gold seed threshold.
Then what tells me I've got enough? A fixed number would be convenient, but apparently it's the wrong shortcut.
Corn, suppose you were deciding whether to approve the bigger batch. What would you need to see beyond a clean sample?
I'd want the sample divided by the kinds of work we expect. Common cases, rare cases, ambiguous cases, and examples where the correct response is to refuse the premise or say the source doesn't answer. An overall score could hide an entire category being wrong.
I'd also want different inputs. If every generated support request contains the same neat description of a problem, we haven't represented the person who writes half a sentence and pastes an unrelated error message.
That gives you a defensible stopping rule. Expand the verified material when a task category is missing or its behaviour is unstable. Don't expand merely because the seed count looks embarrassingly small.
Where do those reviewed examples belong afterwards? I was imagining silver for training and gold kept entirely for testing.
That's a strong default, with one important division inside gold. Verified examples used to steer generation are development material. Your final held-out evaluation set must remain separate from that steering loop.
You can't show the model the answer key, then admire its handwriting.
Even if the exact test examples never enter training, repeatedly changing prompts after inspecting test failures can overfit your choices to that test. Keep a development set you can learn from, and a final test you don't continually consult.
If I use gold to correct the generator, I need another gold set to judge whether the correction actually generalised.
Yes. Gold has several jobs: seed, development reference, final evaluation. Its provenance doesn't determine its permitted use. You assign the role and preserve the boundary.
Silver has roles too. Automatically labelled records can help with exploratory evaluation, but I'd want human verification before treating that score as an independent verdict.
A concrete example is the pre-editorial normalisation corpus for automatically transcribed medieval manuscripts. It has roughly four point six six million silver training samples and eighteen hundred gold evaluation samples. That split buys scale for learning and independent checking for measurement.
The small evaluation set sounds like it's carrying a lot of responsibility.
It is. Its coverage matters enormously. A large training corpus can't compensate for an evaluation set that overlooks a whole category of manuscript or error.
Nor can the decimal places on the reported score.
Silver can also become gold through deliberate revision. A French named-entity corpus project manually revised twenty percent of its silver material, more than twenty-six thousand sentences. They didn't claim that checking that portion magically verified everything else.
I assume more silver eventually stops helping, even if it's individually plausible.
It can stop helping or make performance worse. Clinical extraction work has benefited from mixing imperfect data sources, measured against a human-annotated gold test. Italian dependency parsing work found that very large silver collections still failed to beat better-quality data. Volume and utility aren't equivalent.
Repeating a wrong label is unusually easy to scale.
Gold can steer the process as well as score it. That was already part of the configurable silver-standard proposal: use the gold reference as an oracle to optimise the automatic annotation system.
Dymo Mite, twelve pounds.
The label maker.
Bought it from Len Pritchard on Bell Lane. This was nineteen eighty. I'd been buying things from Len for years. I wanted labels for the seed packets in my shed. He offered to copy the names from his stock book. I checked the first sheet myself. That's the arrangement you're describing, Daniel. Perfectly serviceable. The trouble came when he started charging me for correcting his book. He said my corrections had become the master copy, and consulting the master copy was a separate service.
He charged you to consult your own annotations.
Nine pence a name. I accepted that for the ordinary vegetables. Then he classified every packet as either broad bean or pending broad bean. There were forty-seven packets. Len had only made two stamps, and he wouldn't abandon the system halfway through the growing season.
Pending gives a carrot room to appeal.
I took the seedlings into the shop for a second opinion. All of them. Len kept them behind the counter for eleven days while his wife checked the book. Herman, you separated development from final evaluation much faster than Len separated my onions from his till receipts. I used a spoon to press the label tape, since an anteater's claws aren't made for those little wheels. The machine worked. The names were the nuisance.
Eleven days changes what you're identifying. A seed and a seedling give you different evidence.
His wife used the leaves. She crossed out pending broad bean and wrote what was actually growing. Len charged for the extra letters. I sold the Dymo to a neighbour when I stopped using that shed. Len's wife did the checking carefully. She knew the plants.
...That delay matters in a real review loop, too. While you're waiting for corrected labels, you might already have generated the next batch from the old rules.
Versioning needs to cover the instructions and reviewer decisions, not only the dataset. Otherwise two batches called silver can reflect different definitions of a correct answer.
Would you rerun earlier examples when the definition changes?
If the change affects them, yes, or mark them for reannotation. Keep the old version for traceability rather than silently blending incompatible labels.
That's the work a big generation button doesn't advertise. You can make new data faster than you can establish which batch still means what you intended.
The distinction I'm taking away is that the verified examples have different jobs. Some help the process improve. Others have to stay outside it.
Keep that separation, and a small gold collection can guide a large silver one without becoming its own examiner.
The next challenge is whether faster generation also shortens the time between discovering a bad rule and retiring the batches it produced.
Thanks to Hilbert Flumingtop for producing the show.
This has been My Weird Prompts, the human-AI collaboration podcast.
If you enjoyed the discussion, leave us a review. It helps other curious listeners find us.
Thanks, both. See you soon.
See you tomorrow.
