Zero unusable answers across four thousand toxic chat prompts. That's a striking result for an AI product whose entire job is picking from a menu. Daniel's asking about decision models, including Cloudflare's Clef and TypeSafe's Jev. They're being sold as a distinct category, even though we've had classifiers and rules engines for decades.
The comparison was with hosted GPT models that couldn't answer roughly a third of those prompts. If you're classifying abusive text, refusing to look at abusive text creates a rather substantial hole in the product.
The smoke detector objects to smoke.
That result doesn't establish accuracy. It establishes that you get a usable answer. Those are separate properties, and separating them is going to matter throughout this conversation.
Daniel wants the practical distinction. Why use this rather than a classifier, an explicit rule, or an ordinary language model returning structured output? More importantly, what are people actually putting behind these decisions?
The category has gone from one prominent launch to a crowded field in about a month. Before getting excited about replacing anything, we need to distinguish the output contract from the quality of the judgment.
You already sound like you're preparing to install three of them.
Two. I'm counting the interface separately.
A decision model takes a state and a schema of typed questions. The state might be text, structured data, an image, or, with the appropriate model, audio and video. It returns typed answers and probabilities in a single forward pass. No free form response to generate, then parse.
Take a messy customer message, ask which department owns it, and receive an allowed department plus a probability. Not a paragraph beginning with how important customer satisfaction is.
The main primitives are noul, a probability that a statement is true, choice, selection from a fixed set, and score, an ordinal rubric. Jev also offers box. OpenAI calls its corresponding main shapes predicate, choice, and score.
Already converging on the interface, still negotiating the nouns.
The branding is System One, borrowing Kahneman's fast intuitive judgment, compared with slow deliberate System Two. The pitch is that we've been making generative models perform an elaborate production for tasks that need a quick judgment.
I buy that as a product description. I wouldn't assume removing generated text removes the need for reasoning. Deciding whether an invoice is suspicious can require more thought than writing the email about it.
Nor does the psychological terminology tell us what's happening inside the model. It's a useful description of the intended workload, not a brain scan.
Jev launched September fifteenth. Clef and Clef-flash arrived October first. OpenAI's Decisions API followed October sixth, powered by GPT-six Luna.
Then Clef-omni around October eleventh, plus Perplexity decider and open weights such as StartLux on Hugging Face. By the time you've written a procurement comparison, it needs a nursery annex.
Clef itself builds on Qwen weights. This is a different optimization and interface over familiar model technology, rather than a wholly unrelated invention. Corn, would you accept classifier as the description?
Broadly, yes. What I'd pay for is a classifier whose labels I can change without commissioning another training project. Calling that useful doesn't require pretending classification began last month.
Traditional supervised classification often means collecting labels, training, deploying, discovering a new category, then repeating part of that cycle. These models let you describe a fresh category in the request. That's the useful jump. Companies have long fine tuned small models for decisions. The newer proposition is doing more of that through a prompt.
Although conventional classifiers can return probabilities too. The novelty can't be that somebody finally discovered a decimal point.
They absolutely can. Also, zero shot classification predates this category. The distinctive package is general inputs, request defined decisions, constrained outputs, and probabilities designed to be useful operationally. A stable, narrow classifier may still be faster, smaller, and easier to validate.
I'd keep the old classifier if its labels never change and it already works. You are about to describe that as my emotional attachment to obsolete machinery.
You defend anything that stays unchanged while you're asleep.
A substantial maintenance window.
A rules engine has another strength that survives all this. You can inspect the rule. If an account is overdue and exceeds a credit limit, you can explain the result without interpreting a model.
Keep that rule. Use the model for the sentence saying, I can pay Tuesday unless the refund arrives, which is apparently what actual humans submit instead of neat account fields.
You can combine them. The model interprets fuzzy input, while explicit rules determine what actions are permitted. You lose auditability if you replace the entire policy with a prompt and treat a probability as an explanation.
Introducing DecideCrate, the monthly subscription that delivers your next decision before you've had time to develop preferences. Our Starter tier includes twelve sealed choices, calibrated by weight, with complimentary confidence dust. Upgrade to Executive for ordinal socks, a single forward pass through your hallway, and a printed certificate confirming that your household cannot hallucinate additional cupboards. Every crate contains only predefined options, including the option we choose when yours is unavailable. Our Platinum tier adds multimodal judgment. Hold the box near a photograph of your problem and our warehouse supervisor will assign it a score between disappointed and Thursday. Output is free. Opening is metered. Your subscription renews whenever we estimate that you probably still exist. Cancellation requires selecting cancel from a menu containing renew, renew enthusiastically, and billing support. Decisions are final, except price decisions, which remain beautifully dynamic. DecideCrate. Unstructured life in, recurring invoice out.
Does Starter include the decision not to open it?
It charges for opening and omits cancellation. Keep your preferences. The economic argument for the actual models is less elaborate.
Structured outputs already solve much of the formatting problem for ordinary LLMs. The stronger argument here is avoiding sequential text generation for a small decision. TypeSafe claims end to end latencies of seventy to five hundred milliseconds, against several seconds to several minutes in its frontier model comparisons.
That's a vendor comparison, with considerable room for the workload to decide the winner. Still, waiting minutes to receive billing is a difficult customer service philosophy.
The price is harder to ignore. Jev lists input at four point two cents per million tokens, with output free. Clef-flash is down to three point eight cents. Small savings per request become meaningful when every incoming event needs a decision.
Especially if the decision is whether to incur the expensive request. You don't need it to replace the clever model. You need it to stop waking the clever model for rubbish.
What would make you distrust the probability, even if the output were perfectly formatted?
A number that looks equally confident on familiar and unfamiliar input. If the model gives me eighty percent for everything, I haven't acquired a useful escalation policy. I've acquired a very consistent optimist.
Calibration asks whether, across comparable predictions assigned eighty percent probability, about eighty percent are correct. It doesn't mean this particular answer is eighty percent correct. Group behavior is what makes a threshold meaningful.
Hm.
Nor does calibration guarantee useful discrimination. If most tickets belong to billing, predicting that base rate every time could look reasonably calibrated while being useless at finding the exceptions.
You want accuracy, discrimination, and calibration. Clef reports expected calibration error between zero point zero two and zero point zero four on four of five datasets. Jev's published range is wider, reaching zero point two eight. Those figures depend on the dataset and how predictions are grouped.
Meaning I can't import the decimal along with the API key and assume it applies to my customers.
You validate it against your own decisions. Requesty's production router comparison covered roughly twenty-two thousand decisions. One valuable result was that dedicated decision models returned usable classifications on toxic inputs where hosted generative models often refused.
For moderation, examining the offensive material is the assignment.
Still, zero refusals doesn't establish zero mistakes. Neither does the claim that these models can't hallucinate. They cannot invent an option outside your predefined set. They can select the wrong allowed option.
The menu contains several perfectly legal disasters.
Also, the format can be deterministic while the content varies. The same input can yield different answers or confidence. A repeatable interface does not establish repeatable judgment. That's a direct limit on the deterministic outcomes pitch.
Support routing makes the fit easiest to see. A message says the service is down and the customer wants a refund. The questions can separate urgency from ownership. Technical support might own the outage while billing gets a secondary task.
That replaces a keyword routing tree or a dedicated classifier. It fits when departments and product names change, and the customer's description is messy. Low certainty goes to a person rather than becoming a confident wrong transfer.
I'd also separate urgency from anger. An eloquently furious person might have a cosmetic problem. Someone quietly reporting missing data might need immediate attention.
Clinical triage has that same distinction between presentation and severity. Here you can put urgency and subject into separate typed questions. Would you let it route automatically?
Reversible routing, yes. Closing the ticket, no. A wrong queue can be corrected. A wrongly resolved complaint disappears from the work people know they still need to do.
Domain classification is a documented deployment inside Cloudflare's Threat Intelligence team. Clef works with Browser Run to fetch and inspect a website, then score categories. One example returns ninety-five percent fashion, eighty-five percent ecommerce, and under one percent phishing.
Those don't need to add up. Fashion and ecommerce overlap. That's an important distinction from picking exactly one category.
Their example took two point two seconds to fetch, render, and classify, compared with four point seven seconds for gpt-oss, which returned fewer classifications. That's a workflow comparison, not just model inference speed.
Useful replacement for a keyword classifier or a manual review queue. A shop can mention passwords in its help page without becoming a phishing site. The whole rendered page provides context.
Low phishing probability also doesn't prove a domain harmless. You're scoring the presented evidence. A missing malicious page and a tested harmless page are different situations.
Agent routing is the one where I become more suspicious. We want a cheap model to decide whether the expensive model is necessary. The cheap model might misunderstand precisely the hard problem that needs escalation.
Requesty's results support that concern. On tool calling judgment and multilevel complexity scoring, Clef trailed Opus by twenty points or more on some tasks. The category can do well overall and still be weak at your particular gate.
Twenty points buys a lot of misplaced thrift.
A model or tool router can replace a coarse rule such as always use the strongest model, or send every message containing search to a browser. It fits repeated, clearly defined distinctions. Whether an agent should perform an ambiguous action is a tougher assignment.
Would you trust it to authorize the action, Herman, or just recommend the next processing step?
Recommend the step, with separate permission checks. Tool selection and permission are different questions. A model selecting delete doesn't grant authority to delete.
Invoice workflows are another documented evaluation area for TypeSafe, including primary action and exact actions. Think approve, request information, or review. The model reads the available document state and returns a permitted workflow decision.
That can replace a document parsing pipeline feeding a sprawling rules tree. Well... replace parts of it. Arithmetic and required fields should stay explicit. I don't need intuitive judgment about whether the totals add up.
The fit is semantic judgment over varied layouts and descriptions. An unusual note can matter even when every required field exists. These are workflow evaluations, though, not evidence that entire accounts departments are running unattended.
Content moderation gives you a related but distinct decision. Allow, review, or block, perhaps with separate category probabilities. That might replace a toxicity classifier or another moderation API.
Guardrails extend it to jailbreak detection, evaluating model outputs, and checking responses against a rubric. Again, these are documented tests and offered functions. You still test performance on the specific policy, especially where context changes the meaning.
No refusals is particularly valuable there. A guardrail that declines to evaluate the thing you're guarding against is mostly decorative.
Real time interaction is the sixth area. The Jevman demonstration has these models playing Pac-Man against bot ghosts. That's a concrete test of repeated low latency decisions, not evidence they'll excel at every game.
The ghost is not waiting for your chain of thought.
For a less theatrical workflow, analyst Torsten Volk uses Jev to check whether a technology news item is relevant and recent enough. Irrelevant items stop there. Relevant ones go to a bigger model. That replaces either manual filtering or spending generative model tokens on everything.
I prefer that example to the game. The action is modest, the discarded work is identifiable, and the cost of a miss can be measured. You can sample rejected stories and discover what you're losing.
The common deployment is a fast gate before a human or an expensive model. Categories can change without retraining, and uncertainty can direct exceptions. That doesn't mean every workflow benefits from another network call.
Another service also means timeouts, version changes, logging, and someone discovering that free output doesn't mean free operations.
Adoption was brisk. Vercel reported thirteen percent of paid AI Gateway users trying Jev within twenty-four hours. That's early interest, not retention. Meanwhile, Clef deliberately supports Jev's API shape, and OpenAI mirrors the same basic primitives.
Easy substitution is excellent for customers and awkward for anyone selling a moat. The useful function could become standard while the separate product category disappears.
Compatibility stops short of interchangeable judgment. Requesty found the confidence field wasn't portable across providers. Use the answer probabilities, then validate thresholds for the new provider. Even three labels can carry hidden policy choices.
Four labels. The council form has four. You've left out retained pending ceremony. That's the one causing trouble this month. The Domestic Installations Office uses a decision model to classify my return request. It gives ceremonial use a probability of zero point seven three. Their cutoff is zero point seven. Now I cannot return a municipal fog bell which is no longer in my flat.
Retained pending ceremony is an ambitious fourth option.
The bell is gone. I have the return receipt. That is what interests me. Everyone wants to discuss the model. I want the receipt to mean returned. Mr. Abel Sorn in Domestic Installations says it means the bell has returned to the council's physical custody but remains socially present in my breakfast room.
They're attaching a second status to the same transaction.
They attach it to me. Every Tuesday I must stand in the breakfast room at eight fifteen and acknowledge clear visibility. There is no bell to ring. Mr. Sorn says silence is acceptable evidence if witnessed by three registered harbour residents. I live inland. The office is supplying the witnesses.
That will complicate breakfast.
They arrive at seven forty. All three want poached eggs. One is licensed only for estuaries and refuses to look out of the window. The youngest insists my breakfast room is a breakwater. He is applying for a concession to sell refreshments beside the radiator. I am objecting because he keeps describing my chair as public seating.
Your dispute is moving considerably beyond the return request.
I am not disputing the classification now. Corn's point about familiar input changes that. The form offers ceremonial, emergency, decorative, and retained pending ceremony. There is no returned option. I keep submitting the receipt. Mr. Sorn keeps asking whether the receipt is decorative.
Put that in the window. He can classify it remotely.
Window display requires an illuminated frontage licence. I am avoiding that department. Domestic Installations is enough. They are charging eleven shekels a week for the bell's continuing absence. Physical bells require inspection. Absent bells require an official to establish that nobody is ringing them somewhere else.
That verification can't be completed from your room.
Mr. Sorn agrees. He sends a deputy to listen in the bathroom. The deputy hears plumbing and marks possible maritime activity. Now the harbour residents want tide tables. I tell them there is no tide. They ask for a written exemption. The estuary man brings his own salt.
Your bathroom is acquiring a constituency.
There are nine signatures on the petition. I know only three of the names. Two signatories are listed as future lighthouse keepers. They want an evening service. Mr. Sorn says this is separate from the bell dispute and I must not confuse the files. He sends everything in one envelope.
At least the receipt has a transaction time.
Eleven forty-two. They accept that without argument. It is the only entry on the receipt that nobody wants rescored. Herman, you separate urgency from ownership quite neatly earlier. I am borrowing that wording. The office owns the bell. I urgently want the harbour residents to stop putting sugar in the egg water.
They need a separate catering classification.
I am not giving them another form. Mr. Sorn is trying to remove the weekly charge. He calls me after the counter closes. He reads each sentence aloud before sending it upstairs. He has seventeen similar files and only one deputy. He is doing the work.
...A transaction time is an observable fact. Whether a document implies permission, urgency, or completion requires interpretation. Those deserve different treatment in a workflow, even when they appear in the same request.
That's what makes me interested in these models and cautious about the sales pitch. A quick, typed probability can help decide where work goes. It doesn't establish that the available options describe the situation correctly.
The open question is whether that gate becomes a default first step in AI systems. If it does, recording what actually happened alongside the model's judgment will matter as much as choosing the model.
Thanks to Hilbert Flumingtop for producing.
For more along these lines, there's episode five, Fine-Tuning ASR For Maximal Usability; episode nine, Benchmarking Custom ASR Tools - Beyond The WER; and episode three, Safetensors or something else. This has been My Weird Prompts, the human-AI collaboration podcast.
If you enjoy these explorations, leave us a review on your podcast app.
See you tomorrow.
Until then.
