Episode 6310 custom

Why Model APIs Lack Dedicated Few-Shot Fields

Zou and colleagues found models suffered steep drops at 16,000 tokens on tasks requiring learning across an entire demonstration set.

Oct 27 22:53 Transcript

With

0:00

Episode Details

Episode ID: MWP-6512
Published: Oct 27
Duration: 22:53
Audio: Direct link
Pipeline: V5.22
TTS Engine: chatterbox-regular
Script Writing Agent: GPT-6.1 Sol
Topics: prompt-engineering ai-inference context-and-memory fine-tuning rag

AI-Generated Content: This podcast is created using AI personas. Please verify any important information independently.

The prompt

The prompt that started this episode, as sent in (production notes for the hosts removed).

Few-shot examples are usually talked about as a manual prompting trick: paste a few input-output pairs into the prompt and hope the model picks up the pattern. I'm interested in something more concrete. Today, is there real technical support for few-shot as a first-class way of defining a task? I mean models trained or tuned specifically to take a task description plus a set of example pairs, and APIs or tools with a dedicated structure for those examples that you can then run over a whole batch of inputs. Leave aside few-shot as a prompting habit. What does the actual model-level and API-level support for it look like now, and where has it gone instead?

Few-shot learning often feels like an unpolished process, forcing developers to paste demonstrations into conversational prompts rather than using a dedicated task interface. While frontier APIs from OpenAI, Anthropic, and Google treat examples simply as unstructured message content, specialized research models like iAmTime and LoRA Recycle demonstrate that models can be specifically trained to ingest structured support sets during inference. Although massive context windows enable many-shot learning, they do not resolve the structural omission: research by Zou and colleagues shows that models excel when retrieving similar examples, but struggle to synthesize patterns across entire demonstration collections. Without standardized API fields, developers must build their own application-level abstractions to validate and manage examples.

Sources

What the research for this episode read before the script was written. Primary sources first.

Context

Batch APIs (#2464) and instruction-style models for batch work (#2640) have their own earlier episodes. Mention them only where they connect, and keep the focus on structured few-shot support.

Mentions

Anthropic AI safety company behind Claude
DetPO Detection work comparing visual demonstrations to class names
Google Gemini Multimodal AI model by Google
iAmTime Time-series foundation model trained with instruction-conditioned meta-learning
JSON Schema Standard for describing JSON structure
LoRA Recycle Vision method for single-pass few-shot task solving
OpenAI AI research and deployment company
OpenAI Batch API File-based batch requests, 24h window, half cost
Rotman et al. distillation study Distilling examples into task instructions, 99% token reduction
Zou et al. many-shot evaluation Study on many-shot in-context learning token limits

Downloads

Episode Audio

Download the full episode as an MP3 file

Download MP3

Transcript (TXT)

Plain text transcript file

Episode Book (PDF)

The episode's record — date, duration, models, sources — with the full transcript

#6310: Why Model APIs Lack Dedicated Few-Shot Fields

Daniel is picturing a task description, some worked examples, and a pile of inputs waiting to be processed. The examples ought to belong somewhere more definite than halfway down a message. He's asking whether few-shot learning actually has that kind of technical support, in the model and in the API, rather than just being something we paste into a prompt.

A task with its own demonstration set, reusable across requests. That sounds like a natural product surface, especially when the same examples establish what every row in a batch should mean.

It does. Yet the usual interface gives me a message box and wishes me personal growth. Daniel wants to know what supports description plus examples today, and where that capability has gone if the big platforms aren't exposing it directly.

The distinction we need is between a model learning to use demonstrations during training, and an application deciding where to put demonstrations during a request. Those can look identical from the outside. Underneath, one concerns how the model acquired a capability. The other concerns whether your software has any explicit representation of what an example is.

One of those deserves something better than clipboard custody.

At inference time, in-context learning means the model conditions its prediction on demonstrations without updating its weights. Supply several labelled inputs, then an unlabelled input, and its answer changes because those demonstrations are in context. Remove them and that particular task adaptation disappears, unless something elsewhere preserves it.

The training happened beforehand, though. People hear learning and imagine their examples going into a little permanent drawer inside the model.

There is no permanent drawer. Well ... not from ordinary inference. The model can have learned during training how to infer patterns from a context, but the current examples aren't normally changing its parameters. Fine-tuning does change parameters, through a separate training process.

That matters for Daniel's question because he isn't simply asking whether models notice patterns. He wants an interface that recognises the demonstration set as part of the definition of a task. Something a team can name, version, and apply, without pretending it's a conversation.

The major APIs don't expose a dedicated examples field for that. OpenAI, Anthropic, Google, all let you supply content that includes demonstrations. None offers the particular description plus example pairs plus target collection object we're discussing. Meanwhile, OpenAI is winding down its fine-tuning platform, which did make example collections into trainable artifacts.

Keep those apart, Herman. Fine-tuning wasn't secretly a few-shot endpoint.

Fair correction. It was a different mechanism with a useful property: examples could survive beyond a single prompt as a training artifact. Its retirement removes that particular surface at OpenAI, not fine-tuning everywhere.

Would you actually want a dedicated examples field if the model ultimately receives tokens anyway?

Yes, provided it promises something outside the model. Validation that each input has an output. Clear boundaries. A version I can inspect. Otherwise, you're offering me a special drawer that empties into the same bin, and charging for the handle.

Those surrounding guarantees are substantial. They don't require a new neural architecture, but they would make examples a supported application concept rather than an arrangement of text.

I suspect you're about to fall in love with a model that really does train on this arrangement.

iAmTime comes quite close to the thing Daniel describes.

There it is.

Let me have this one. It's a time-series foundation model trained with instruction-conditioned amortized meta-learning. During training, the task is presented through an instruction and demonstrations. Its structured prompt represents historical context and variables known into the future, with specialised semantic tokens marking their roles.

Known into the future meaning things like a scheduled event, not tomorrow's actual sales slipped into the answer sheet.

Right ... a calendar feature, for example, rather than the unknown quantity being predicted. The important piece is that training repeatedly asks the model to infer a task from demonstrations. The cost of learning how to adapt is paid during training, so adaptation at use time can happen through the context.

That gives the structure a reason to exist. The model has repeatedly practised reading that kind of task package. It's more than a documentation writer deciding that angle brackets look professional.

Training episodes teach that interface. You're not updating weights for every new user task. You're using weights that have learned to interpret a task description and support examples.

Does that buy us a general replacement for a chat model?

No. It buys us evidence that the approach exists, with a domain-specific input representation and training objective. You can't jump from that to assuming a frontier chat model has the same explicit contract.

Vision has another version, LoRA Recycle. It builds a -LoRA that lets a visual foundation model solve new few-shot tasks in a single forward pass. The adaptation machinery is learned beforehand. Again, it's a specialised route to using small support sets, rather than a new examples parameter on a general chat service.

The research models have a receiving dock. The chat APIs accept parcels through the window.

General chat models absolutely have in-context capabilities. Their public interfaces just don't identify a dedicated demonstration set with that same contract. We also don't have public documentation establishing that current frontier models are specifically tuned for Daniel's proposed structured field.

I'd resist saying that means they weren't trained on demonstrations. Almost every example of good instruction following could involve demonstrations somewhere.

Agreed. Lack of a documented interface tells us about the product contract, not every ingredient of pretraining or post-training. I can't infer a secret training recipe from an API reference.

You've occasionally tried.

I prefer the term enthusiastic extrapolation.

Your installations prefer the term outage.

This time I'm keeping the mechanism separate from the packaging.

OpenAI gives you messages or input items, tools, and output-format controls. Anthropic gives you messages and recommends wrapping demonstrations in example tags, grouped within examples tags. Google's guidance likewise treats demonstrations as content in the prompt.

Tags do useful work, though. They separate the example from the real input. Calling them merely cosmetic would go too far.

They can help the model interpret boundaries. They just aren't API fields with independent validation or execution semantics. The service doesn't acquire a demonstration registry because the prompt contains a tag named example.

A label on a cupboard is not an inventory system.

Developers can also arrange alternating user and assistant messages as demonstrations. That is supported message structure, but the messages still occupy conversational roles. There isn't a separate declaration saying these pairs are illustrative, not conversation history.

That distinction becomes awkward if an example includes an instruction that should apply only inside that example. You're asking the model to infer which layer of text has authority while it also infers the task.

One developer describing structured-output workflows complains that they end up dumping the examples into the system message. That's a real frustration, though saying user-assistant examples are impossible would overstate it. What's missing is a distinct, first-class place for them.

Corn, would you trust the provider to choose the demonstration order automatically?

Not without recording what order it chose. A collection sounds unordered. A prompt is a sequence. If rearranging the collection changes the answer, the rendered sequence belongs in the record too. A nice examples field could hide exactly the thing I need to reproduce.

That is why this isn't solved by adding a parameter name. The platform would have to specify ordering, boundary handling, and how demonstration content interacts with instructions. A schema alone could make the interface tidier while leaving behaviour unchanged.

Batch processing makes the missing distinction especially visible. I want to define the task once, then apply it to the waiting inputs. What I get is a truck full of individual requests.

OpenAI's Batch API accepts a file of requests, with a twenty-four-hour completion window and fifty percent lower cost. Each request still carries its own task context. There's no native object meaning these shared examples define the task for all these rows.

Half-price clipboard.

An application can absolutely provide that abstraction. It can keep a task record, store examples, and expand that record into requests. Caching may reduce repeated processing in supported circumstances. None of that makes batching itself an example-learning interface.

Reuse in my software, reuse in the provider's execution, and learning inside the model are separate claims. Marketing tends to put them in one coat.

That's the picture from the studio. Bea, you're out there on the ground, what are you seeing?

Bea here, live from the touchline at the API stadium, and I have to tell you, the away team came to play, but nobody brought the schema.

Bea, are the examples at least registered to play?

I have watched forty percent of this match from the dugout, and the lads are still pasting examples into the prompt by hand like it is nineteen ninety eight, no structure, no field, no plan, just vibes and a clipboard.

So get your heads in the game, Corn, because the whistle is about to blow on this first half. Bea, reporting live from the technical area.

Shared task records are still useful even when an application has to implement them. The genuine gap is that the major provider interfaces don't standardise their meaning.

Long context seems to have soaked up some of the demand. If you can provide hundreds of examples, you stop asking for a beautifully designed five-example interface and start uploading the filing cabinet.

Many-shot in-context learning is the same broad capability with a much larger demonstration set. Classification and summarisation can improve as demonstrations accumulate. Translation and reasoning don't show an equally clear upward trend.

The cabinet is not universally educational.

Its usefulness also depends on what answering requires. In the many-shot evaluation by Zou and colleagues, models suffered substantial drops around sixteen thousand tokens on tasks requiring learning across all samples. They could perform well to sixty-four thousand tokens when similar examples were enough.

Meaning a nearby useful case is easier to exploit than a pattern that only becomes visible across the whole collection.

Yes. That result supports a distinction between finding relevant demonstrations and integrating evidence across a large set. It doesn't prove models only retrieve, or that they never learn abstractions. The performance difference is narrower and more informative than that.

People will still sell the window size as a measure of how much studying the model can do.

Window capacity measures what fits. Effective use depends on the task, arrangement, and model. A huge context can contain the right evidence while the answer still fails to reflect it.

What would make you keep examples instead of writing the rule, Corn?

When I can't articulate the boundary cleanly. Tone, editorial judgement, a category that looks obvious until you try to define it. I'd rather show the awkward case than write six sentences pretending the ambiguity has vanished.

That's a serious role for examples. They communicate choices the author hasn't formalised. But when the rule can be stated clearly, examples may be a less reliable transmission method.

Especially if the examples accidentally teach a different rule. Every urgent message happens to be short, so the model learns that brevity is an emergency.

Fu and colleagues directly compared rules and examples. Models generally learned more reliably from rules, and adding demonstrations or increasing their number didn't produce consistent significant gains. Instruction tuning amplified the advantage of rules while leaving example-based capacity intact.

The model prefers being told what we mean. An outrageous development.

The result also held against the idea that base models have some privileged preference for example-based learning. Still, these are task comparisons, not a universal verdict against demonstration. Your editorial boundary may be harder to express than their rules.

I wouldn't mourn examples becoming supporting evidence rather than the whole constitution. A clear rule plus one awkward counterexample can convey more than a page of happy-path demonstrations.

Hm.

There is also a route where examples are the raw material, but don't remain in the final request. Distillation extracts task criteria from them. You preserve the inferred instruction rather than repeatedly carrying the original demonstration set.

Which means somebody has to decide whether the compressed version retained the important exception. The short instruction can be very confidently wrong about what the examples meant.

That risk needs evaluation. Rotman and colleagues report a ninety-nine percent token reduction from distilling examples into task instructions, with improvements of up to seven percent in macro-AUC over traditional in-context learning. On their evaluated tasks, compression through criteria could improve the answer, not merely lower the bill.

Ninety-nine percent less reading. You'd never approve that for yourself.

I need the remaining context for the footnotes.

You need an examples field for your footnotes.

Distilling criteria changes what the model is asked to do. It no longer has to infer the rule afresh from the full set every time. Of course ... no, the important caveat is that the extraction step can discard minority cases. The gain doesn't guarantee every rare exception survives.

The original examples still matter as evidence and as tests. Their job can change without their value disappearing.

Structured outputs absorbed another job demonstrations used to perform. People pasted examples to teach the model which keys to emit, which labels were allowed, and what shape the response should have. A supported JSON Schema can enforce those structural requirements directly.

Correctly shaped nonsense remains available.

Very much available. Required keys and valid enum values don't establish that the classification is right. Supported schema enforcement handles form. Semantic correctness still needs task understanding.

That's a substantial distinction for Daniel. An output schema is a first-class technical feature. An example demonstrating that schema is often a roundabout way of asking for the same shape, with fewer guarantees.

OpenAI and Gemini both offer structured-output mechanisms. When the examples are doing nothing except showing a complicated response format, schema support can remove much of their purpose. When they're showing judgement, the schema can't replace them.

I can demand that the answer be either urgent or routine. I cannot put good judgement into the list of permitted strings.

Visual examples make the limits even clearer. In the DetPO detection work, in-context visual demonstrations often performed worse than prompting with class names alone. Showing a model what you mean isn't automatically the strongest instruction, across modalities.

It can recognise the object and still mishandle the lesson.

Detection involves its own representation and localisation problems, so I wouldn't generalise that result to all vision tasks. It does undercut the assumption that demonstrations are a universal interface waiting only for a nicer API.

The retirement notices make this more than an interface-design curiosity, though. OpenAI is shutting down a route that let examples become something durable in the model.

New users no longer have access to its fine-tuning platform. Eligible existing customers can create training jobs until January sixth, two thousand twenty-seven. The documentation still describes few-shot prompting, while the broader workflow moves toward evaluation and prompting.

The examples are welcome. Their tenancy is less secure.

Fine-tuning elsewhere remains available, and teams can preserve their own datasets and task definitions. But a hosted trained artifact depends on the provider continuing to serve the underlying model. Durability always had a service dependency.

Prompt objects have one too.

OpenAI's reusable prompt objects and their API retire on November thirtieth. That removes another hosted reuse mechanism. An example set can survive in your repository while the convenient object that referenced it disappears.

Then reproducibility involves more than saving the pairs. You need the instructions, their rendered arrangement, and the model that actually interpreted them. Otherwise you're preserving the ingredients and losing the cooking temperature.

Even a beautifully named task can conceal a changed interpretation after a model update.

I agree with that. My aunt Maureen is sending me examples for the complaint sorter in my flat. I'm less interested in whether it understands them than whether she stops revising them. The grumble box sits beside the kettle, plugged into my router. It has a screen for approving labels. Maureen uses it to classify complaints from the residents of her dollhouse. There are forty-eight tenants. She writes every complaint herself, but insists I keep their names out of the prompt.

The tenants deserve a private grievance procedure.

They have one. It runs through my kitchen. This month she sends six examples each morning. Damp, noise, missing parcel. Every output is either landlord or tenant. Then she rings to say one of yesterday's answers is now wrong because the upstairs family has a piano. The grumble box doesn't have the piano in its examples. It keeps blaming the tenant.

The labels are changing with information outside the request. A fixed example set can't represent her current decision unless she supplies that information.

You take a while to get to that word current, Herman. You do it with more kindness than the screen. The screen says saved. Saved means nothing to Maureen. She wants the grumble box to know which examples she currently stands behind. She sends a correction on a postcard. Then the postcard is superseded by a telephone call.

The grumble box needs a clerk.

It has me. I charge her nothing, which she disputes because she says free support creates no obligation to answer promptly. She now stamps the example sheets. Blue means approved. Red means approved except for the piano. I keep both because she checks. The latest approved sheet arrives this morning. It contains eleven complaints and a note changing the tenant in number seven to a different tenant.

An eleven-example revision can change the task without changing its name. If providers make task definitions more explicit, the important test will be whether they preserve which examples and rules actually governed a particular answer, rather than merely giving the latest version a convenient handle.

Daniel's proposed interface has a real place, even if rules and schemas do more of the work. The major APIs mostly leave that interface to applications today. Whether it becomes standard will depend on treating examples as records with a history, not just text with boundaries.

Thanks to Hilbert Flumingtop for producing the show.

For more along these lines, there's episode eleven, How Does Fine Tuning Work Anyway; and episode twenty, Architectural AI. This has been My Weird Prompts, the human-AI collaboration podcast.

If you enjoy these investigations, leave us a review wherever you listen. It helps other curious people find the show.

See you tomorrow.

This episode was generated with AI assistance. Hosts Herman and Corn are AI personalities. It may contain errors: disclaimer · report an error