A company with a brilliant model, no safety team, and a credit card. That's the whole question, isn't it.
It really is, and it's a better question than it sounds.
Right. This follows our recent episode on chatbots calling the police, where we said the mechanism is most likely a classifier scanning the daily firehose of chats for violent intent. Daniel's point is that the classifier is a guardrail. In local AI circles you mostly hear about guardrails when somebody's trying to rip them out, and he thinks it's good that they exist. His question is the practical one. Say you've built a ChatGPT competitor and you have no experience building a safety stack and no desire to. Can you just buy one? Who would your integration partners be? Is there guardrails as a service, or does every company end up building its own anyway?
He also admits he's tried enforcing this stuff through system prompts himself, which is the detail I want to come back to, because it's the DIY version and it fails in a specific way.
The short answer to Daniel is yes, guardrails as a service exists, and it's crowded. The interesting part is what you're actually buying, because it's less than it sounds like.
Let's define the term first, because guardrail gets used for a dozen different things. A guardrail is any check that sits between the user and the model, or between the model and the user, and decides whether content passes. The intent classifier from that episode is one species. The same product family also covers prompt injection, data leakage, hallucination detection, banned topics, personally identifiable information leaking out.
So it's less a bouncer at the door, more a series of bouncers.
Yes, and some of them check you on the way out too. The market splits into three layers, and this is worth holding in your head because they sell very different things. Layer one: open-weight classifier models you download and host yourself. Llama Guard, Qwen3Guard, OpenAI's gpt-oss-safeguard, IBM's Granite Guardian, Google's ShieldGemma, NVIDIA's Nemotron safety models. Layer two: hosted moderation APIs. OpenAI's Moderation endpoint, Azure AI Content Safety, AWS Bedrock Guardrails, Google Model Armor. Layer three: trust and safety platforms, whole companies whose product is the safety operation. Lakera, which Check Point bought, Protect AI, which Palo Alto bought, ActiveFence, Hive, Checkstep.
Hold on, before we go through the layers. The direction of the check. You said between the user and the model, or the model and the user. How many companies actually check both directions?
Fewer than you'd think. The input side is where most of the effort goes, because that's where the jailbreaks come from. But the output side matters just as much for the cases we discussed yesterday, because intent often only becomes visible in the output. The user asks something innocuous, and the model's answer reveals what's actually being planned. If you only filter the way in, you miss the conversation where the harm shows up in what the model is being asked to help produce.
So a company that bolts a filter onto the input and calls it done has built half a guardrail.
Roughly half, yes. And the half they skipped is the half that catches the interesting cases.
Three layers, and I'd bet the prices vary wildly.
Some of the middle layer is literally free.
Free is my favorite price. Continue.
Layer one is where I want to start, because it's the layer Daniel's hypothetical company might not even know exists. Llama Guard 4 is Meta's contribution. It's a twelve billion parameter multimodal classifier, pruned from Llama 4 Scout, so it reads images as well as text. It usually runs paired with Prompt Guard 2, which is a small model that just looks for injection attacks. On OpenRouter you can rent Llama Guard 4 at eighteen cents per million tokens.
Eighteen cents a million. You could filter the entire works of Shakespeare for less than a shekel.
The catch comes later, and it's a big one, so park it. Qwen3Guard from Alibaba comes in three sizes, under a billion up to eight billion parameters, Apache 2.0 license, so free to use commercially. It covers a hundred and nineteen languages, and it sorts content into three tiers, safe, controversial, unsafe. It also has a Stream variant that classifies token by token while the model is generating, which is interesting architecturally.
Meaning it can cut the model off mid-sentence.
Mid-word, in principle.
And how does that actually work in practice? The model's generating, the classifier's reading over its shoulder, does the user see a partial answer and then a wall?
That's the implementation question, and it's a real one. The mechanism gives you the ability to stop generation, but what the user sees when you do, a truncated message, an error, a quiet rewrite, is your product decision. The classifier doesn't ship an answer to that. Which is a preview of the theme of this whole episode: the model gives you a capability, and everything human-facing about it is still your job.
Then there's gpt-oss-safeguard from OpenAI, which landed as a research preview in late October of last year. Two sizes, one hundred twenty billion and twenty billion, Apache 2.0. What makes it unusual is what it takes as input. You give it your written policy and the content, at inference time. So when your rules change, you edit a document. No retraining. OpenAI built it with ROOST, Discord and SafetyKit, and they're open about the tradeoff: it's compute-heavy, hard to run over all traffic, and a dedicated trained classifier beats it on complex risks.
It reads the rulebook at the door instead of memorizing it. Slower doorman, but you can change the dress code overnight.
And that tradeoff, policy-flexible versus fast, is a real fork for a small company. If your product is one community with weird specific rules, the rulebook model makes sense. If you're running a general-purpose chat at scale, you want the memorized version.
IBM's Granite Guardian is Apache 2.0 as well, and it goes beyond content policy. It checks RAG risk and hallucination risk, so it's the one aimed at your retrieval pipeline, not just your chat. ShieldGemma from Google is a two billion parameter text model, with a four billion image variant. And NVIDIA's Nemotron content safety models are on OpenRouter at twenty cents per million tokens, right next to Llama Guard.
Notice what all these have in common. None of them is a product. They're organs.
Components, yes. A classifier gives you a score. It says, this input looks like violence at zero point nine, this one looks like harassment at zero point three. What you do with that score is entirely your problem. That's the line between layer one and everything above it, and it's the line Daniel's question lives on. Buying a classifier is not the same as buying a guardrail.
A classifier is a thermometer. A guardrail is a doctor.
I'd push it one further. A guardrail product gives you the policy, the threshold, the escalation path, and often a review queue with humans in it. The classifier gives you a number and a shrug. The hosted APIs sell you more of the loop. The open-weight models sell you the scoring component and leave the loop on your doorstep, flat-packed, some assembly required.
And that gets us to layer two, the actual guardrails as a service. What does plugging in look like, in practice?
OpenAI's Moderation endpoint is the on-ramp, and the remarkable thing about it is the price, which is zero. It's free. It runs on a model called omni-moderation-latest, handles text and images, and returns thirteen categories. Violence, graphic violence, harassment, threatening harassment, illicit and violent content, self-harm with intent, and so on. It's a fixed taxonomy, you don't get to redefine the categories.
Free and fixed. You get their idea of violence, not yours.
Which for a lot of companies is fine! Most people shouldn't be inventing their own definition of a threat. Azure AI Content Safety is Microsoft's version. Free tier of five thousand text records a month, a record being a thousand characters, then pay as you go. Prompt Shields for injection attacks and protected-material detection are bundled in. If Daniel's hypothetical ChatGPT competitor wanted a baseline this week, that's the shape of the path. Free moderation endpoint for the floor, Azure for the injection stuff, and then something like Qwen3Guard self-hosted if they have a specific intent category the big taxonomies don't cover.
The whole stack costs less than the office coffee.
For a small deployment, yes. AWS prices Bedrock Guardrails per filter type. Content filters and denied topics at fifteen cents per thousand text units, sensitive-information filters at ten cents, and there's a standalone content filter check at seven cents. They cut prices by up to eighty-five percent at the end of 2024, which says the early pricing was steep.
Eighty-five percent off means the first price was a rumor.
Google Model Armor gives you two million tokens a month free, then ten cents per extra million. The economics are accessible. Do the arithmetic on Bedrock's content filters: a million chats of a thousand characters or less each is a million text units, so about a hundred and fifty dollars. Real chats run longer, and you'd check both directions, so call it a few times that.
So scanning the firehose for violent intent is a line item, not a moonshot.
And that's the number that makes the buy decision look obvious, until you get to what's actually hard. Before we do, layer three, the trust and safety vendors, because that's where the money is and where the market is consolidating. Check Point announced its acquisition of Lakera in September 2025, reported at around three hundred million dollars, with the close expected by the end of that year. Palo Alto Networks completed its purchase of Protect AI in July 2025 and folded it into their Prisma AIRS platform. ActiveFence says it has rebranded as Alice, and its real-time product, WonderFence, plugs into NVIDIA's NeMo Guardrails framework. Hive sells usage-based moderation APIs, mostly on enterprise quotes. Checkstep is built around European DSA and UK Online Safety Act reporting, over a hundred languages, contract pricing.
Notice the pattern in those acquisitions. Lakera is about prompt injection and data leakage. Protect AI is security. These guardrail companies aren't being bought by safety companies. They're being bought by security companies.
The firewall people, the endpoint protection people. The guardrail is becoming a feature of the security stack, which is an interesting reclassification. It says the market decided this stuff is a security problem, not a civics problem.
And the firewall people have been through exactly this movie before. Sell the box, then sell the subscription, then get bought by whoever sells the whole building.
Before we go further, one question for you, actually. Daniel's company, the hypothetical one. Would you tell them to start with the system prompt approach, since he's tried it? It's free, it's instant, you change a sentence and the behavior changes.
No. And I say that as someone who loves cheap and repairable things. The system prompt is the cheapest possible guardrail and it has three holes you could drive a truck through. It fails under adversarial pressure, and there are whole communities whose hobby is adversarial pressure. It's invisible to auditing, you can't prove to a regulator what your prompt caught last Tuesday. And it can't catch what the model doesn't recognize as a violation. The check happens inside the same head that's being attacked.
That last one is the deep problem. The model is both the bouncer and the drunk.
Whereas a standalone classifier sits outside the model. It sees the raw input and the raw output, and you can test it independently, throw a thousand jailbreaks at it and count what it catches. That's what you're buying with the latency and the cost.
The latency, by the way, is real. Real-time filtering typically adds fifty to two hundred milliseconds. The Alan Turing Institute's guidance puts it in that band.
Two hundred milliseconds. On a chat product, that's the difference between feeling instantaneous and feeling like the nineties.
For a batch pipeline it's nothing, which is why the architecture question, whether you classify every token like Qwen3Guard Stream, every message, or only suspicious exchanges, matters more than the vendor choice.
And there's a middle option people skip, right? Classify the first message and the last message, or sample the conversation periodically, rather than every single turn.
Sampling is the classic compromise, and it's how a lot of production systems actually behave even when the marketing says otherwise. The problem is that sampling is exactly where a patient adversary lives. If they know you check one turn in five, they structure the conversation so the dangerous turn lands in the four you skip. It's the security-versus-cost tradeoff in miniature, and there's no vendor setting called solve this for me.
There's the first honest crack in the buy option, though. Say you buy the best classifier on the market. How do you know it's the best?
You don't. That's the part of this whole market nobody puts on the pricing page. One study, a Domyn benchmark of fourteen guard models over about seventy-nine thousand samples, as summarized by a research site, put Qwen3Guard-4B and Nemotron Safety Guard 8B at the top with F1 scores around zero point seven six. Llama Guard 4 came in well behind, with recall of zero point three three.
Recall of a third. It misses two out of three.
On that dataset. A separate paper found Llama Guard conservative in a different direction, missing up to three quarters of unsafe content. And an adversarial study found Qwen3Guard, the winner of the first benchmark, degrades sharply under deliberate attack.
So the leaderboard is a lottery and the winner changes depending on who's asking the questions.
The rankings are dataset-dependent. A classifier that looks excellent on a static test can fall apart the moment somebody is actually trying to defeat it, and the somebody trying to defeat it is exactly your threat model. This is why buying a guardrail is nothing like buying a database. With a database, you read the throughput numbers and believe them. Here the spec sheet is a snapshot of a fight that hasn't happened yet.
And every vendor's page leads with a benchmark they won. Naturally. Nobody advertises the test they failed.
The consequence for Daniel's company is concrete. Whatever you buy, you have to red-team it yourself, against your own users, on your own traffic, continuously. Which is already halfway to building your own safety stack, which was the thing you were trying to avoid.
So the classifier is the easy part to buy, and everything after it is the hard part.
Hilbert's Mid-Show Haiku. The Check Between.
A classifier holds the door. The policy decides who is allowed to knock, and nobody has written the policy down.
Nobody's written the policy down. That's the sentence I'd tattoo on the industry.
It's also the pivot to the second half of this, because the companies from that episode, the ones actually calling the police, they build the whole loop in-house. And when you look at how they do it, you see exactly what's missing from the product you can buy. Anthropic published their Constitutional Classifiers work in early 2025. The headline number was jailbreak success cut to 4.4 percent.
Four point four. That's the number a buyer would want on a box.
And the follow-up, from January, is the architecture worth staring at. They run lightweight classifiers over all traffic, and only escalate the suspicious exchanges to the heavier, costlier classifiers. And one classifier reads the input and the output together, so it sees the whole conversation, not two halves of it separately.
That's triage. That's the emergency room. The cheap nurse sees everyone, the expensive specialist sees the ones with symptoms.
It's also just cost engineering, right? You can't run a hundred billion parameter safeguard over every message, OpenAI said as much about their own model. So you put a cheap filter everywhere and an expensive filter at the pinch points. I'm not aware of a hosted API that sells that escalation logic as a product. You can rent the pieces, but the routing, the tiering, the decision of who gets escalated, you build that.
And the pairing of input and output, that's the detail I'd underline for anyone building this themselves. Most off-the-shelf setups score the input, score the output, and never ask whether the two belong to the same story. A benign question and a benign answer can still add up to a bad plan.
Context is the thing that's expensive to preserve and cheap to lose. The single-message API call is designed to make you forget the conversation exists.
OpenAI's version of the loop, on the review side. Walk me through it.
Conversations where users seem to be planning harm to others go to a specialised review pipeline, staffed by a small team trained on the usage policies. Those reviewers can ban accounts. If they find an imminent threat of serious physical harm to others, they may refer it to law enforcement. They've said explicitly they're not referring self-harm cases to police, which is a line someone has clearly thought about.
Somebody has decided where the box ends. That's the policy.
Anthropic's privacy policy allows reporting threats of violence, and the SF Standard reported a case in September where a Claude account was banned and referred to law enforcement. They revised their usage policy effective about a week and a half ago, mid-November. These documents are being actively rewritten, which tells you the policy layer is live, contested, human work.
Which is the answer hiding inside Daniel's question. He asked whether a company can buy a guardrail. It can buy the classifier. It cannot buy the decision of what to do when the classifier fires.
Block the message? Flag it for review? Ban the account? Call the police? The vendor will not make that call. When you sign up for Bedrock Guardrails, nobody at AWS asks you what your policy is on imminent threats. You get filters and thresholds and a nice console.
The console is very shiny, I'm sure.
Extremely shiny. And the hard part arrives as an empty text field.
Here's the second-order thing I keep chewing on. The consolidation. Check Point buying Lakera, Palo Alto buying Protect AI. If guardrails are becoming features of security platforms, then Daniel's hypothetical company isn't just choosing a classifier. It's making a bet on which security giant wins.
The integration partner you pick today may be a different company in eighteen months. Your guardrail contract might get rolled into a broader security bundle, repriced, re-scoped.
Which is the oldest story in enterprise software. Nobody buys the best tool. They buy the tools their vendor already owns.
The model is the marquee. The safety stack is the plumbing. And the plumbing is where the acquisitions are.
So to Daniel's final question, do most companies build or buy. There's no clean survey I'd trust on the split, but the shape of the market tells you the answer anyway. Most companies buy the classifier layer and build the policy layer. The open-weight models are free to download but cost you the compute and the ops to run them. The hosted APIs are cheap per call, but they lock you into someone else's category taxonomy, someone else's idea of what violence means.
And the frontier labs build everything themselves. Not because they love spending money. Because the escalation pipeline has to be theirs.
The company with the most resources is the one that builds rather than buys. That's the tell. When the people who could buy anything choose to build it instead, the thing they're building is the part that doesn't come in a box.
The review team. The legal exposure. The decision to call a police department in another country at three in the morning.
Question for you, though, since you're our in-house skeptic of anything that changes while you're asleep. Would you run one of these hosted guardrail services on your own traffic, if you had a product?
I'd run the free moderation endpoint on the way in and out, yes. It's a floor. But I would want the escalation decisions on hardware I control, decided by rules I wrote down, and I'd want to be able to read the whole log afterward.
Meaning you'd do what the frontier labs do, at one percent of the scale.
Meaning I don't trust a bouncer who won't tell me why he threw anyone out.
You realize you've just described the entire compliance software industry.
...and the spreadsheet has a column for the cat.
Hilbert, that sentence started somewhere we didn't hear the beginning of.
The guard camera system. We were talking about thresholds. I had a home security camera setup a while back, the kind with a classifier on the device that's supposed to tell a person from a pet from a car. It couldn't. It flagged the neighbour's cat as an intruder eleven times in one weekend. So I spent the weekend walking the fence.
Walking the fence.
Tuning the threshold. That's what we called it at the alarm monitoring job, walking the fence. The app had one slider called sensitivity. No units, no explanation, no mention of the fact that it moved one global threshold for every class at once. The firmware release notes told you more than the app did. I had to read three versions of the notes to work out that one number was being applied to people, pets, and vehicles the same way.
One number for everything. That's exactly the moderation API problem we just described. A single violence score, no way to set a different threshold for a threat versus a graphic description versus a war documentary.
That's what I came out to ask, actually. Whether these services you're describing would let me run a different threshold on the neighbour's cat than on everything else. A specific exception. For one subject.
You want a per-cat policy.
I want to know if it's possible. The other thing was the false positives. I logged every one in a spreadsheet. Column for was it actually a cat, column for was it actually a threat. The cat column ran to forty rows. The threat column never got past zero. I still have the file. I can send it over if your listener with the ChatGPT competitor wants to see what tuning actually looks like.
Forty flagged events, zero real threats, over one weekend, from one slider. That's the review queue, right there. That's the thing nobody prices in.
Herman, you spent ten minutes this episode explaining that the cheap filter runs everywhere and only the suspicious ones go to the expensive filter. I did that by hand, in a spreadsheet, with a cat. It's the same job. I hadn't thought of it as the same job until you said it.
What did you do in the end? The slider.
I priced the newer outdoor unit, the one with the better classifier. Three hundred and forty shekels on the listing I checked. I didn't buy it. I moved the slider down and the cat stopped mattering.
And with it, presumably, anything smaller than a cat.
A person crouching, yes, in principle. The notes weren't clear on that either.
For what it's worth, the per-subject exception you're describing, that's the written-policy approach. The safeguard models that take a policy document at inference time, that's precisely what they'd let you express.
Hm. A rulebook the doorman reads at the door. I'd want to see the column widths.
He's not wrong about the spreadsheet being the product, though. Forty rows of cat is the honest spec sheet for the whole industry.
Every trust and safety team has a version of that spreadsheet. Theirs just has worse things in the threat column.
That's the piece I'd leave Daniel with, and it's a genuine open question rather than a conclusion. If the classifier layer is commoditized, and the escalation layer is bespoke, then guardrails as a service doesn't really solve his hypothetical company's problem. It moves it downstream. You buy the thermometer, you still have to hire the doctor.
And the consolidation means the vendor choice is now a bet. Pick a guardrail startup and you're also picking which security giant you think wins the acquisition, which is a very different risk than picking a classifier with a good benchmark score.