Four point two billion parameters. That's the size of the model that decides which frontier model your question gets sent to.
And it never writes a word. Not one.
It reads your question, it reads the candidate answers, and it hands back a number. That's the whole job. Which is exactly the thing Daniel wants to poke at this week, because he's noticed something strange about a whole family of models sitting on Hugging Face under the text-to-text label that don't actually do text-to-text in the way anyone means it.
Right, and he's making a sharper point than that. He's saying calling them non-text-generating is a misnomer. They're deeply textual. They read text, they embed text, they classify text. They just don't speak. And then he wants the use cases, the agentic harnesses where these things run quietly under the hood, and the difference between models that return tabular data and models that return structured outputs. He's pointing at two model cards specifically.
The first is Decision 1.0, the Nox model, which the Semantic Router people use for LLM model selection. The card opens with, and I'm quoting, "Your move. Give Nox a state, questions and possible answers. It returns typed decisions and probabilities, with labels defined at runtime." Which is a great piece of model-card writing and also a completely different genre of software than a chatbot.
And the second one is TF Decision Trees, sitting in Hugging Face's tabular data section. Gradient boosted trees, binary classification on structured data. Trained on the US Census Income Dataset, roughly three hundred thousand instances, forty-one variables, predicting whether somebody makes over fifty thousand a year. Two models, both small, both silent, and they could not be more different in how you actually use them.
So today we're looking at the models that don't talk back. The ones that just quietly decide. And I think the place to start is the word Daniel used. Non-text-generating.
It's the wrong word, and it's wrong in an interesting way. What these are is discriminative models, not generative ones. That's the real axis. A generative model produces a distribution over the next token. A discriminative model produces a distribution over a label. Both of them are doing statistics on text. Only one of them is trying to talk to you.
The Semantic Router paper puts it in information-theoretic terms, which I actually think is the cleanest version. A bidirectional encoder is maximizing mutual information between its hidden states and a task label. A causal decoder is optimizing next-token prediction. The encoder is squeezing the input for what it tells you about the answer. The decoder is squeezing the input for what comes next in the sentence.
Understanding versus producing. And once you see it that way, the whole category stops being weird. A sentiment classifier isn't a broken chatbot. It's a completely different animal that happens to eat the same food.
Same input, different output shape. Text goes in, a label comes out.
And the scale here is worth sitting with. Impulse2000's sentiment transformer is about thirteen million parameters. Dl26's Veyra-100M is encoder-only, a hundred million. Those are rounding errors next to anything anyone calls a frontier model. And then Nox, the big one we're going to talk about, is four point two billion. Which sounds enormous until you remember it's the thing choosing which nine-hundred-billion-parameter model gets to answer you.
The small model is the bouncer. The big model is the band.
And the bouncer doesn't need to play guitar. It needs to know who belongs inside.
So let's do the two ends of the spectrum properly. Nox first, because it's the more interesting machine. Four point two zero eight billion parameters, adapted from Qwen3.5-4B, Apache 2.0, sixteen thousand token budget for the complete question.
And the thing that makes it unusual is that sentence on the card. "Returns typed decisions and probabilities, with labels defined at runtime." That's three output types doing three different jobs. Choice, which routes a request or picks among two to two hundred and fifty-five actions, and returns a selected ID plus a distribution. Noul, which checks a condition against supplied evidence and returns the probability the condition is true. And Score, which applies two to ten ordered rubric descriptions and returns an expected index plus a distribution.
So it's not answering a question. It's scoring options.
It's scoring options and it's telling you how confident it is about the scoring. And the runtime part is the new thing. The candidate answers aren't baked into the weights. You supply them in the request. So today you ask Nox to route a query between billing and technical. Tomorrow you ask it to route between urgent and low-priority. Same model, same weights, no retraining, no fine-tuning, no new model card.
That's the part I keep circling. A traditional classifier is trained to output one fixed set of labels. This thing takes the label set as an argument.
And that's a real architectural commitment, not a marketing line. The candidate head reads candidate endpoints and the final query vector together. So the candidates are part of the computation, not just part of the prompt. One forward pass per question, batched in groups of eight. And critically, no explanation tokens. It doesn't generate reasoning and then extract an answer. It just returns the answer.
Which is why the latency is what it is. There's no chain of thought to wait for. There's no thinking out loud.
There's no thinking out loud at all. It's the least chatty software in the building.
Give me the benchmark numbers, because I think they're more interesting than the headline.
Seventy-two point eight four percent overall accuracy across three thousand seven hundred and sixty-six scored decisions and fifty-four tasks. It leads Kev-4B by two point seven five percentage points. The bigger sibling, Lux, is nine billion parameters off Qwen3.5-9B and scores seventy-six point seven two, which is plus four point eight three over Kev-9B and plus three point eight eight over Nox.
So doubling the size buys you about four points.
About four points overall, and the breakdown is where it gets interesting. The overall score is weighted. Decisions thirty percent, Composition twenty-five, Reading fifteen, Inference fifteen, Transfer fifteen. And those weights were chosen after they saw the results. The card is honest about that, which I appreciate. It says reweighting is not a training improvement.
Meaning they picked the weights that flattered the model.
They picked weights that reflect what they think matters for the product, and they say plainly that this happened after observing outcomes. That's a disclosure you rarely get.
And on the underlying components?
Nox scores eighty-three on Decisions. Lux scores eighty-three point two one. Basically identical. The gap is all in Composition, where Lux gets fifty-one point eight eight and Nox gets fifty-one point seven nine, so also basically identical, and then Reading, where Lux gets ninety point three one and Nox gets seventy-nine point zero six. That's an eleven-point gap on a single sub-skill.
So the smaller model is just as good at making the call and much worse at reading the material it's making the call about.
Which is a strange and specific failure profile. It's decisive and slightly illiterate.
That's a lot of people I've met.
And it matters for deployment, because the Reading weight is only fifteen percent in their composite. If your actual use case is document triage, that eleven-point gap is the whole ballgame and their headline number is hiding it from you.
Which is the general lesson about every benchmark table ever printed. The weights are a choice.
The weights are always a choice. Now, how does the Semantic Router actually use this thing?
This is the part I want to get right, because it's the clearest production example of small models running under the hood that I know of.
The router describes itself as a symbolic Mixture-of-Experts gate with deterministic early exit. That's the architecture in one phrase. It uses thirteen signal types. LoRA-based classifiers for domain, complexity, jailbreak detection, PII detection, modality, factual grounding, user feedback. Each of those is a small non-generative model doing one narrow job.
So before your question ever reaches a large model, thirteen small models have already looked at it.
And the router's own model-selection module is a second layer of the same thing. KNN with quality-weighted voting. SVM with an RBF kernel, gamma of one point zero, running on seven hundred and eighty-two-dimensional normalized embeddings. An MLP that's a feedforward net, ten thirty-eight into two fifty-six into one twenty-eight into N. The feature vector is a thousand twenty-four-dimensional Qwen3 embedding plus a fourteen-dimensional category one-hot.
So a thousand and thirty-eight numbers describe your question, and then a small feedforward network decides where it goes.
And Nox is the decision model that makes the final selection. So you've got a stack of small models feeding a medium model that routes to a large model. It's routers all the way down.
The latency numbers tell you why anyone bothers. Heuristic signals are sub-millisecond. The machine learning signals run ten to a hundred and twenty milliseconds. Demand-driven evaluation cuts total signal extraction latency by fifty to seventy percent.
And that's the entire economic argument. You spend ten milliseconds to avoid spending two seconds on the wrong model.
Now, the hardware constraint, because this is odd and I want it on the record.
Nox and Lux are validated only on AMD gfx942 under ROCm. CPU and MPS are unsupported. NVIDIA is described as unqualified. On a Hugging Face release.
In a world where NVIDIA is the default assumption for everything, a model card that says NVIDIA is unqualified is practically a political statement.
It's a validation statement, not a political one. They tested on what they had and they're telling you what they tested. But you're right that it's unusual, and it means most people reading that card cannot run it on the machine they own.
And then the caveat at the bottom, which I think is the emotional core of this whole episode.
"It evaluates supplied evidence without live retrieval; confidence does not guarantee correctness."
Say that again slowly.
Confidence does not guarantee correctness. The model will hand you a probability distribution, and that distribution is a statement about the model's internal state, not a statement about the world.
So a router can be ninety-five percent sure and wrong.
A router can be ninety-five percent sure and wrong, and it will route your question to the wrong model with total composure, and the wrong model will answer it with total composure, and the user will get a mediocre answer with no error anywhere in the logs.
Nothing crashes.
Nothing crashes. That's what makes it nasty. A misconfigured router doesn't throw an exception. It just quietly sends everything to the cheap model, or sends all the hard questions to the small one, and the only symptom is that the answers are slightly worse than they should be and nobody can prove it.
Which is a failure mode you can't page anyone about.
You can't page anyone about it, because there's no incident. There's just a slow drift in quality that everyone attributes to something else.
That brings us to the other end of the spectrum. TF Decision Trees.
keras-io slash TF Decision Trees. Gradient boosted trees for binary classification of structured data, implemented in TensorFlow 7.0 or higher. And the mechanism is worth walking through because it's completely different from anything neural.
Start with the data.
The US Census Income Dataset. Roughly three hundred thousand instances, forty-one numerical and categorical variables. The task is binary. Does this person make over fifty thousand dollars a year, yes or no.
And the encoder?
They implement a custom Binary Target encoder as a Keras preprocessing layer. It encodes the categorical features with respect to their target value co-occurrences. Then those encoded features build a decision forests model.
Which is a fancy way of saying it looks at which categories tend to show up alongside the high earners and turns that into a number.
That's exactly what it does. And the evaluation is ninety-six point five seven accuracy with a validation loss of zero point two two seven on Census-Income.
Ninety-six and a half percent. That's a spectacular number.
It's a spectacular number on a dataset where the base rate does a lot of the work, and I'd want to see the notebook before I trusted it as a general claim. The card doesn't say how target leakage was handled, and a target encoder is precisely the kind of preprocessing step where leakage creeps in and inflates your accuracy.
So you're saying the ninety-six is real but the ninety-six might be flattering.
The ninety-six is what they measured. Whether it generalizes to a fresh census sample is a separate question the card doesn't answer. And here's the number that actually tells the story: thirty-two downloads last month. Not deployed by any inference provider.
Thirty-two.
Thirty-two.
So this model has essentially no users.
It's a reference implementation. It exists to demonstrate a technique, and the technique is the point, not the model. But that's also the honest picture of the tabular data section on Hugging Face. It's full of well-built demonstrations that nobody ships.
Which is a shame, because gradient boosted trees on tabular data are still, in a lot of domains, the thing that actually works.
They're still the thing that wins on most tabular problems. Neural nets have spent a decade trying to take that crown and mostly haven't.
Now the contrast Daniel actually asked for. Tabular data versus structured outputs. And I think the cleanest way to say it is this: one is a fixed instrument, the other is a general-purpose decision engine.
TF Decision Trees has forty-one columns baked into training. It knows about age, workclass, education, marital status, occupation, and thirty-six other things, and it knows nothing else. You cannot ask it a question about a forty-second variable. The schema is frozen in the weights.
And Nox takes the schema as an argument.
Nox takes the schema as an argument. You hand it a state, a question, and a set of candidates, and it scores those candidates. The candidates can be anything. Billing versus technical today, urgent versus low-priority tomorrow, and the weights don't change.
So the trade is obvious on its face. The fixed model is more accurate on its one task. The flexible model is less accurate on any given task but can do all of them.
And that's the real trade, and it's not a close call either way. It depends entirely on whether your problem is stable. If you're predicting census income, you want the tree. If you're routing queries in a system where the categories change every quarter, you want the decision model, and you accept the accuracy hit as the price of not retraining.
Which is the same trade as specialized versus general labor anywhere.
And the same trade as a jig versus a 3D printer. The jig is faster and better at the one thing. The printer can make anything and is worse at all of it.
Now, the user experience difference, because Daniel asked about that specifically and I think it's underrated.
There's a real difference in what it feels like to integrate these two things. TF Decision Trees returns a class label and a probability. You get a number between zero and one, and you get a prediction. That's it. The interface is a column of floats.
Nox returns typed decisions.
Nox returns typed decisions. A Choice returns a selected ID and a distribution over the candidates. A Noul returns a probability that a condition holds. A Score returns an expected index over an ordered rubric plus a distribution. So the output has structure. It tells you what kind of answer it is, not just what the answer is.
Which changes what the calling code looks like.
It changes everything about the calling code. With the tree, you get a float and you write your own threshold. With Nox, you get a distribution and you decide what to do with the tail. And the distribution is the interesting part, because a distribution over two hundred and fifty-five candidates is a map of the model's uncertainty, and most people integrating it will just take the argmax and throw the rest away.
Which throws away the only thing that would have told them when the router was confused.
Right. The distribution is the diagnostic. If you collapse it to a single ID, you've built a system that cannot tell you when it's guessing.
That connects straight back to the confidence caveat. The information you need to catch a misrouted query is in the output you're discarding.
Which is the same mistake people make with every classifier they've ever deployed. Take the label, bin the probability, move on.
Now, the cogeneration question, because I want to handle this carefully.
Handle it carefully is right, because I think there's a genuine ambiguity here and it's worth naming. In the agentic harness literature, cogeneration does not mean small models running under the hood. The term comes from a paper on agentic program repair, and it means co-generating two artifacts in one trajectory. A fix and a bug reproduction test, produced together in the same patch.
It's about the agent producing two things at once, not about small models being quietly invoked.
It's about the agent producing two things at once. And the reason it's a useful term in that context is that generating the test alongside the fix keeps the agent honest. You can't write a fix that passes a test you also wrote to be trivial.
But Daniel's underlying question is still a good one, which is where do these small models actually live in a running system.
The underlying question is excellent, and the answer is the Semantic Router, and it's a better example than the term he reached for. The router's DSL compiles down to agent orchestration frameworks like LangGraph and OpenClaw, to Kubernetes artifacts, and to protocol gates like MCP and A2A. So the routing layer is not a sidecar. It's compiled into the deployment.
Which means the small models are in the infrastructure, not in the application.
They're in the infrastructure. The application developer writes a query and gets an answer. Between those two events, thirteen signal classifiers and a model-selection module and a decision model have all run, and none of them wrote a word.
Here's the thing I keep coming back to. The router is itself a small-model zoo.
It is. KNN, KMeans, SVM, MLP, plus thirteen LoRA classifiers, plus Nox. That's twenty-odd small non-generative models in one system, and the whole point of the system is that they're invisible.
The real story isn't cogeneration. It's that the most sophisticated routing layer in open source is built almost entirely out of models that don't talk.
That's a counterintuitive fact about where the field is. The flashy stuff is generative. The load-bearing stuff is discriminative.
The talking is the demo. The deciding is the product.
The deciding is the product, and it's made of gradient boosted trees and support vector machines and four-billion-parameter scorers that return a number and shut up.
The paper frames the whole thing as uncertainty reduction, which I think is the right frame. Before any analysis, routing entropy is maximal. Log base two of K bits, where K is the number of candidate models.
Routing is a process of collapsing a uniform distribution over K models down to a point, and every signal you extract removes some entropy.
The question that leaves hanging is whether you're reducing uncertainty or just relocating it.
That's the question. Because the router reduces its uncertainty about which model to pick, and it does that by trusting a confidence score that doesn't guarantee correctness. So the uncertainty doesn't disappear. It moves from the router to the user, who now has no idea whether the answer they got came from the right model.
You pushed the entropy downstream.
Hilbert: Threshold.
Sorry?
Hilbert: You keep saying decision. They're thresholds. I spent two years calibrating thresholds on sorting machines, optical sensors over a conveyor, and nobody in that plant ever called it a decision. You set the cut, the arm fires, the part goes left or right.
What were you sorting?
Hilbert: Bearings, mostly. Some of it was washers. You'd get a batch in and the whole line would drift, so you'd walk the cut up or down until the reject rate looked right. Dumb as rocks. Right ninety-nine percent of the time. The whole plant ran on them.
You're saying there's a difference between that and what Nox does.
Hilbert: There's a difference between a threshold and a probability. A threshold is a promise. You set it at a number and you know exactly what happens at that number. A probability is a guess with a decimal point on it. And the trouble with the guess is you can't walk it up and down, because you don't know what it's guessing about.
That's the calibration problem exactly. A threshold you can verify against outcomes. A probability you have to trust.
Hilbert: We had a rule. Anything under eighty-five percent confidence went to a human inspector. And the machines were never wrong when they were confident. But they were confidently wrong about two percent of the time. That two percent is what got people fired.
Confidently wrong.
Hilbert: Confidently wrong. You'd have a part come down the line, machine says good, inspector signs off, part goes out, comes back six weeks later. And the machine was still confident. It was confident the whole time.
There's no way to audit that after the fact, because the confidence score is the only record and it says everything was fine.
Hilbert: The score says everything was fine. The customer says otherwise. And you're standing there with a printout that agrees with the machine.
How did you catch it?
Hilbert: You didn't catch it. You found it. Somebody downstream opens a box and there's a bad part in it, and then you go back through the batch and you find out the sensor had drifted a hair and nobody noticed because the confidence never moved.
The confidence never moved.
Hilbert: Never moved. That's the thing about a threshold. When it drifts, the number drifts with it. When a probability drifts, it still reads ninety-four percent and it's still wrong.
Which is precisely the Nox caveat. Confidence does not guarantee correctness.
Hilbert: Whoever wrote that card has been on a factory floor. Anyway. I've got to go. My sister's expecting me.
The anteater thing.
Hilbert: She's got a thing at four.
The misconception to kill before we go. The one people carry into this whole category.
It's that a confidence score is a warranty. People see ninety-four percent and hear "ninety-four percent likely to be right." What it actually means is "ninety-four percent likely to be the answer this model was trained to give," which is a statement about the model, not about the world.
The correction is one line. A confidence score measures the model's certainty, not the model's accuracy, and those two things can drift apart for years without anyone noticing.
Which is why the audit question is the real one. If these small decision models are running under the hood of every agentic harness, and their confidence doesn't guarantee correctness, how do you audit a router? How do you know when it's silently sending your hard questions to the cheap model?
I don't have an answer to that. The router reduces its own uncertainty and hands the residue to the user, and the user has no instrument to detect it.
The entropy doesn't vanish. It just moves somewhere nobody's looking.
That's the one to sit with. Thanks to Hilbert Flumingtop for producing, and for the sorting machines.
This has been My Weird Prompts. If you've got a weird prompt, send it in. Email us at show at my weird prompts dot com.
We'll be back soon.
