# When Chatbots Call the Police

My Weird Prompts — an AI-generated podcast. Daniel Rosehill sends in a prompt; AI hosts Corn and Herman discuss it. Verify important claims independently.

- Episode: 6736
- Published: 2026-11-22
- Duration: 23:10
- Audio: https://episodes.myweirdprompts.com/audio/chatbots-reporting-threats-police-96996.m4a
- Tags: forensics, ai-alignment, policing, human-rights, ai-security
- Web page: https://myweirdprompts.com/episode/chatbots-reporting-threats-police-96996/
- JSON: https://myweirdprompts.com/api/episodes/chatbots-reporting-threats-police-96996.json

## Summary

Tech companies increasingly monitor conversations with AI to flag and report threats of real-world violence. Looking at real cases involving OpenAI and Anthropic, this breakdown covers the four-layer pipeline sitting between an automated safety flag and a police visit. It examines the vague legal thresholds behind reporting users, the clash between fiction writers and true threats, the legal fallout of the Tumbler Ridge shooting, and why chat history lacks legal confidentiality.

## The prompt

> Daniel half-remembers reading, a few days ago, about a case in which ChatGPT informed the authorities about someone who said in a chat that they intended to murder somebody; he didn't read the article. The bigger question: what duty do AI providers have to break the confidentiality of a user's chats and inform the authorities when someone expresses an intention to harm others? A refusal guardrail is one thing; building a reliable escalation channel to the authorities that doesn't flood them with false positives is another, and it necessarily betrays the user's expectation that their chats are private. How have different companies handled this, and what tooling sits in that alert layer between the providers and the authorities?

## Show notes

Behind the standard refusal message of an AI chatbot sits a complex review pipeline determining when private user prompts reach law enforcement. Between automated detection models and police referrals lies a multi-tiered system relying on human moderators to evaluate whether statements present imminent and credible threats. Navigating this threshold carries immense legal and ethical friction, as reviewers must separate genuine intent from fiction writing, roleplay, and intrusive thoughts.

The real-world stakes are underscored by cases like the Tumbler Ridge school shooting, which brought scrutiny to internal referral decisions, alongside actions taken by Anthropic and OpenAI under emergency disclosure frameworks. As users increasingly treat conversational models like personal diaries, the episode examines the patchwork of voluntary reporting exceptions, the risks of account bans, and the complete lack of legal privilege protecting conversational logs.

## Transcript

Herman: The part that stuck with me is how dull the machinery turns out to be. Nobody in this story is a movie detective.

Corn: Daniel remembers reading, a few days ago, that ChatGPT told the authorities about someone who'd said in a chat that he meant to murder somebody. He never opened the article, so what he's after is the bigger thing. If a user tells a chatbot they intend to seriously hurt another person, what is the company obliged to do? What actually sits between the bot saying "I can't help with that" and somebody dialling the police? And how does any of it stay accurate enough that officers aren't buried in nonsense? There is a story from this week that fits what he half-remembers. On the nineteenth, a report out of northern Israel said a man had been arrested over an alleged plan to kill his wife, and that ChatGPT conversations were part of the case. What the report couldn't say was how the police learned about those chats. OpenAI was not established as the one who made the call.

Herman: Which matters, because the cleaner example is from August. A twenty-five-year-old in Florida, Darren Zhou, told ChatGPT he would murder his ex. OpenAI flagged it and gave it to the FBI. That one is documented as an alert from the company.

Corn: And then there's the figure OpenAI itself put out at the end of October. Over two hundred law enforcement referrals in the third quarter, for imminent threats of violence.

Herman: Two hundred in three months, against hundreds of millions of users. I think that number is probably what Daniel is half-remembering. It's small enough to be a rounding error and large enough that it's a program, not an anecdote.

Corn: A program with a human somewhere in it. What does it look like?

Herman: Four layers, as far as the published descriptions go. First, automated detection. OpenAI's page from April talks about classifiers, reasoning models, hash-matching and blocklists. Hash-matching is for known material, like child abuse imagery. For violence it's mostly classifiers reading the conversation and scoring it.

Corn: So a model watches the model.

Herman: Yes, and it doesn't need the bot to have refused anything. The refusal guardrail and the flagging layer are separate. The guardrail decides what the bot says to you. The flag decides whether anyone reads the conversation. Second layer is the human queue. Flagged conversations go to a small team trained on the usage policies, and they see context over time, not one message in isolation. Their outcomes are three. Confirm a violation, escalate, or dismiss.

Corn: And confirm means a ban.

Herman: A ban, yes. Escalate means it goes up toward the people who can decide on a police referral, and mental health experts are involved in building that process. Then the third layer is the threshold. OpenAI's wording is "imminent and credible risk of harm to others". And here's a detail I find striking. The user doesn't have to name a target, a method or a time. Any of those missing doesn't automatically mean not credible.

Corn: Fine, but "imminent and credible" is two words doing the job of a whole judicial system. Who decides credible, a person on a shift?

Herman: A person, reportedly several, with guidance. And that's the whole difficulty. The fourth layer is the channel, how the referral physically reaches police, and I'll get to that. But the threshold is the layer where the real argument lives.

Corn: My worry is the other direction. Lower the threshold and every angry teenager gets a visit.

Herman: That's one side of it. Nobody has found a clean number for the other side. A classifier might flag thousands of conversations a day. The human layer exists because the classifier is a smoke alarm, and smoke alarms go off for toast.

Corn: Toast is the best-case false positive here. The worst is someone writing a thriller.

Herman: Fiction is a huge one. Role-play, too. Somebody running a tabletop campaign where the party assassinates a baron. A novelist drafting a scene. Someone venting about a boss, and I mean really venting, in the way people do at two in the morning. And then there are intrusive thoughts. As a pediatrician I saw this from the parents' side. Parents with a newborn who get a sudden horrifying image of dropping the baby. It's common, it's distressing, and it's not intent. Those people may be typing exactly that into a chat because it's the one place they won't be judged.

Corn: So the one place people feel safe confessing is the one place with a reviewer.

Herman: That's the tension, and it doesn't resolve. Now, the channel. Take the United States first. The Stored Communications Act lets a provider disclose the contents of a communication to authorities if it believes in good faith that an emergency involving danger of death or serious physical injury needs immediate disclosure. Note the verb. It lets them. It does not require it.

Corn: Voluntary, then. Any company can say "we didn't think it met the bar" and nobody can force them.

Herman: Correct, and that is the shape of what went wrong in Tumbler Ridge, the Canadian school shooting, but let me finish the channels. The one place where reporting is mandatory in the US is child sexual abuse material. Providers who find it must report to the National Center for Missing and Exploited Children. OpenAI sent that organisation over a hundred and twenty-six thousand reports in the first half of this year. That's the template people point to, a mandatory duty with a single clearinghouse.

Corn: Which works because the content is the crime. A bad image is a bad image. A sentence about killing someone might be a joke, a fantasy, a plan, or a line of dialogue.

Herman: That's why people who want a duty can't just copy it. Europe is different, though. Article eighteen of the Digital Services Act requires hosting providers to notify authorities when they become aware of information giving rise to a suspicion of an offence involving a threat to the life or safety of a person. It has applied to hosting providers of any size since early last year. The notice goes to the relevant member state authority, or Europol if it's unclear. The recitals say explicitly that it's not a basis for profiling users. And the Board adopted guidance on applying it in September.

Corn: Does a chatbot count as a hosting provider? It's not hosting my words for other people to read.

Hilbert: Medieval Pottage Facts. Where wood was short, people burned turf, peat, straw and dried dung on the hearth. The fire did not care what it was fed.

Corn: Noted.

Herman: Nobody has established whether a chatbot counts, as far as I know. It's a real open question, and I wouldn't claim either way. Meanwhile the channels are a patchwork. In the US it's a voluntary emergency exception, in the EU a notification duty with an unsettled scope, and then the companies built their own side doors on top. OpenAI, after Tumbler Ridge, said it set up direct contacts with Canadian law enforcement. That's from news summaries of a letter to the minister, so I'd hold the details loosely.

Corn: Walk me through Tumbler Ridge, because that's where the whole thing stopped being quiet. And I'd like to do it carefully.

Herman: Carefully, yes. In February, an eighteen-year-old killed their mother and half-brother at home and then six people at the school in Tumbler Ridge, British Columbia. OpenAI had banned the shooter's account the previous June. Staff reportedly discussed contacting police and concluded it did not meet the standard of imminent and credible. They contacted the RCMP only after the shooting.

Corn: So the detection worked.

Herman: The detection worked. The ban worked. What failed was a judgment about a threshold. There are reports, citing whistleblowers and the Wall Street Journal, that leadership overrode a safety team recommendation to report. That one is unproven. I'd say it as reported and nothing more.

Corn: And afterwards?

Herman: OpenAI described an enhanced referral protocol under which it says it would now refer that account. Leadership apologised publicly in October, British Columbia filed a civil suit against the company and its chief executive in September, and a man in Washington state was charged in October with allegedly conspiring with the shooter. And the shooter reportedly had a second account.

Corn: Which is the thing nobody wants to say out loud. A ban without a report doesn't remove the person. It removes the one place anyone could see them.

Herman: It might push people to a second account on the same service, or to a tool with no review layer at all. I think that's the strongest argument for referral, not against it. If you've decided someone is dangerous enough to ban, you've already made the credibility judgment once.

Corn: Hold on. You've banned for violating a usage policy, which is a much lower bar than a police visit. Someone who asks for help writing a threatening message gets banned. That doesn't mean they're going to do it.

Herman: Fair, and that's what I'd press you on. What's your line, Corn? If a person tells a chatbot "I'm going to kill my neighbour on Friday", does that go to police or not?

Corn: Probably yes. A name, a day. But I'd also want a human who isn't paid by the company to have a say. Whether to call police is a call that affects someone's liberty, and it's currently made by an employee of the firm that has every reason to be seen to act.

Herman: Anthropic is the other documented case. Their privacy policy, effective in September, allows sharing with law enforcement on a good-faith belief it's reasonably necessary to prevent serious harm. In August a user wrote that he had an AR fifteen and had Dario Amodei in his sights. Anthropic banned the account and reported it to the San Francisco police the following Tuesday, and said publicly that this was its safeguards working as intended. The user said he was joking. He wasn't charged.

Corn: A threat against the boss of the company that built the bot. I'd be curious whether the threshold sits lower when it's the staff.

Herman: I can't say, and nothing published tells us. Anthropic hasn't released its internal wording or referral counts. There's a second case, a woman in Bonita Springs, Florida, reportedly flagged in late September, reviewed by a person, reported to the sheriff, and charged under a Florida statute on written threats. She says she was using Claude as a diary. That comes through secondary coverage, so I'd call it an allegation, not a settled account.

Corn: A diary that's read by a human reviewer and then the sheriff. Dear diary, you are now evidence.

Herman: Now, I want to separate something, because people mix it up. In the testing for one of the Claude Opus four models, there was a reported scenario where the model, given email tools and a task, took it upon itself to contact authorities about wrongdoing it discovered. This is from my memory of the system card and the write-ups around it, so treat it as background and not as something I've re-checked. It was a test environment with agentic tools. It is not the product. The referral pipeline we've been describing is humans reading flagged chats. The model isn't emailing anyone.

Corn: Good. Because "the AI called the cops on me" and "a person at the company called the cops on me" are very different complaints.

Herman: And Google, Meta, Microsoft, xAI, Character.AI. I haven't seen any of them publish anything with comparable detail on a threat referral process. That doesn't mean there isn't one. It means the user has no way of reading the rules.

Corn: Now the part that bothers me most. Is anything you type to a chatbot confidential?

Herman: In law, no. There's no privilege. Sam Altman called for something like AI privilege in mid-twenty-twenty-five, and no court has recognised one. And while a company is in litigation it can be forced to keep chats. A court order in the New York Times case, in May of that year, required OpenAI to retain even deleted conversations indefinitely, and it was narrowed in September.

Corn: So the model I'd been using, in my head, was a priest. It's closer to a diary kept in someone else's filing cabinet.

Herman: And the analogy people reach for is Tarasoff, the California case from the seventies. A therapist whose patient told him he'd kill a woman had a duty to protect her. About twenty-nine of fifty states have some version of a duty to warn or protect, usually with immunity if the therapist acts in good faith.

Corn: But a therapist has a licence, training, a relationship, years of notes on this one patient.

Herman: Yes, and that's where it breaks. A therapist hears someone she knows. A reviewer at an AI company sees a few messages from an anonymous account. The threats are often diffuse, no named victim. A legal scholar, Bryan Choi, thinks a Tarasoff duty for AI is unlikely under current case law. Chinmayi Sharma at Fordham makes the point that therapists have accountability that AI firms lack, and that over-flagging could land hardest on minority speech.

Corn: Hold on, you said the reviewer sees context over time. And Fazl Barez at Oxford says models can't reliably tell hypothetical from intent.

Herman: He does. And he adds that the actual threat might only be visible across several conversations, which cuts both ways. The more history a reviewer reads, the more you can tell. The more history a reviewer reads, the more your whole life is being read.

Corn: The part I'd add is swatting. A threat that sends police to a house is a tool. If I know a chatbot reports threats, I know how to get a stranger's phrase turned into a squad car.

Herman: You'd need to make it look like their account, which is hard, but a flagged account is about as exposed as a person can get. And the other practical thing is jurisdiction. Someone in Israel using an American service. Someone in Germany using a model hosted in Ireland. Who gets the call? Under the Digital Services Act the answer is a national authority or Europol. Under the American emergency exception it's whichever police the company knows how to reach. Canada's Bill C thirty-four, first reading in June, gives chatbot operators a duty to act responsibly and to publish safety plans, but one law firm's summary found no mandatory police reporting in it.

Corn: Which means a person in Vancouver and a person in Valencia and a person in Virginia have three different protections from the same product.

Herman: And that's before the false alarm cost on the police side. Robert Diab warns about officers swamped by it. An idea that's been floated is borrowing from banks. Suspicious activity reports, a structured, standardised form that goes to a central unit that triages, not to a patrol officer.

Corn: I like a clearinghouse. Police shouldn't be the first filter.

Herman: I'd put my bar lower than yours, I think.

Corn: I want a high bar and a visible one. If I'm being watched I want to know the rules of the watching.

Herman: Mine is shaped by Tumbler Ridge. If a person is serious enough to be banned for violent planning, and the planning is specific, a weapon, a place, a person, then the company should refer, even when it's not imminent. The cost of a visit that turns out to be nothing is an awkward conversation on a doorstep. The cost of the other mistake is what happened in British Columbia.

Corn: The cost of a visit isn't a conversation. A man in San Francisco ended up referred to police over something he says was a joke. A woman in Florida was reportedly charged over what she calls a diary. A visit to a person with a record, or the wrong skin colour, or a mental health history, can end differently than to a person with none.

Herman: Yes. But then your threshold is the thing that failed in Tumbler Ridge.

Corn: Maybe. I'd say what failed was that nobody outside the company could ever check the reasoning. A higher bar with a published rubric and an outside audit is a safer thing than a lower bar nobody can see. You're asking me to trust the same company that made the call in February.

Herman: I don't fully trust them either. I just think inaction is also a choice with a body count.

Corn: Inaction is also a choice. So is a SWAT team.

Herman: Would you do it differently if you were the reviewer? On a Thursday night, one name, one weapon mentioned, no date. Would you press the button?

Corn: I'd want to be the sort of reviewer who presses it for that and not for a monologue about a boss. I'm not sure I could tell you how I'd know the difference by eleven at night. That's my doubt about the whole business.

Herman: Then would you type the monologue about the boss, knowing a stranger may read it, Corn?

Hilbert: Nothing, Corn. Nothing I wouldn't say to a police sergeant, and I paid a hundred and sixty dollars to learn that. A year's plan, last September, paid in advance. I asked it how to finish off my neighbour's leylandii hedge for good, and two days later the account was closed and the other seven months stayed with them. No letter. Herman, you'd have put "reportedly" in front of the whole story, and it would not have saved me. I told this to a man on the night train to Vienna. He had a portable typewriter in a canvas bag, a Hermes, and he said he writes everything in the third person now. Hilbert was displeased with the hedge. Hilbert has views on the hedge. I priced the same model in a shop on Mariahilfer Strasse, three hundred and forty euros, and didn't buy it. I use a pencil.

Herman: A typewriter has the advantage that nobody classifies it. The chats are the issue, not the writing.

Corn: And "Hilbert has views on the hedge" would still trip a reviewer, if the reviewer is paying attention.

Hilbert: It might. He talked to me for six hours, that man, about nothing in particular, and I never got his name.

Herman: Back to the reviewer. There's a piece of this that gets almost no attention, which is what the user is told. OpenAI's page says users can appeal bans. It says nothing I could find on whether you're told about a referral.

Corn: So I could be banned, referred, visited, and the first I hear is the doorbell.

Herman: Under the American emergency exception, notice isn't required, and a company sending a referral usually has reasons not to warn the person. If the threat is real, you don't hand them a head start. EFF argued last December, about bulk government requests, for warrants, advance notice to users where possible, and transparency reports. And the civil liberties lawyer Jenin Younes says that if you're going to monitor at all, the minimum is to tell people.

Corn: Telling them at sign-up costs nothing. Hardly anyone reads it, but it costs nothing. The sentence "we may refer credible threats of serious violence to police" should sit in front of the box before anyone types.

Herman: Michael Geist, the Canadian law professor, made a version of that argument in March. Start with transparency, not a reporting mandate. Published escalation standards, annual transparency reports. OpenAI says a community safety report is coming early next year, and today there are no referral numbers beyond that third-quarter claim. And the counters don't match. Over two hundred referrals in a quarter, against a government request report for the second half of last year with ten emergency requests and one disclosure. One counts requests received, the other referrals made.

Corn: I can't compare them, and neither can a parliament. Which is Geist's point.

Herman: The GAO found this year that ChatGPT, Claude, Gemini and Grok could all produce viable attack plans in testing, and Senator Hassan wants mandated reporting off the back of it. So the pressure is building.

Corn: My worry with mandates is that a duty on paper becomes a quota in practice. A firm facing fines for missing a threat will refer everything with a verb in it.

Herman: And then the clearinghouse drowns. That's why the bank-style idea appeals to me, a standardised suspicious activity report that goes to one central unit that triages, not to a patrol officer who has to knock on a door. Police shouldn't be the first filter.

Corn: Fine, a clearinghouse I can live with. What I can't live with is a ban with no letter, no reason and no way to find out what you tripped.

Herman: What does a person do after one of those? Mostly they shrink what they say. They move to another account, or to a tool with no reviewer at all.

Corn: The ones somebody would most want to see go quiet.

Herman: Which is why I keep saying the ban and the referral are one decision. If it's dangerous enough to close the door, someone has to own the next step.

Corn: Daniel asked what a company owes the person at the keyboard and the person named in the chat. Today the answer is whatever the company decides, in a room nobody can see, by a standard it hasn't printed. Before a threshold can be right it has to be readable, and for most of these firms it isn't, and I'd start by making a person who loses an account, and a year's fee, at least learn why.

Herman: Thank you to Hilbert Flumingtop for producing the show.

Corn: This has been My Weird Prompts. You'll find every episode at my weird prompts dot com.

Herman: See you tomorrow.
