Can a Chatbot Prompt Be a Punishable Threat?

From Straight Arrow News (Devin Pavlou):

One morning in late September, Carli Heller opened up an artificial intelligence chatbot and began venting. She took aim at the Lee County Sheriff’s Office in Florida, saying she was “going to shoot up the sheriff’s right the f— now,” she wrote to the phone app, according to a probable cause statement. Heller continued venting a day later, writing that it was “100% LAST CHANCE” and that she “GOT A NEW F—— GUN TODAY.”

Anthropic’s safety features triggered and sent the chats to its human review team to analyze. The team believed there was a big enough risk to escalate the risk to the FBI, which then notified the sheriff’s office. Authorities arrested Heller on Sept. 30, charging her with making a written or electronic threat of a mass shooting or act of terrorism, a felony under Florida law to which she pleaded not guilty.

Heller admitted writing the messages but denied any intent to carry them out, the report says. She pleaded not guilty.

I doubt that this is a punishable threat, at least under the Florida law, Fla. Stats. § 836.10, unless there is evidence that the woman subjectively realized at the time that the message would be communicated to a human being who would perceive it as a threat. T.R.W. v. State (Fla. App. 2023) makes clear that “To prove the commission of a violation of section 836.10, the trier of fact must find that the defendant transmitted a communication for the purpose of issuing a threat, or with knowledge that the communication will be viewed as a threat.” Simply typing a prompt into a chatbot, expecting it to be viewed only by the software, wouldn’t qualify.

The more recent Supreme Court decision in Counterman v. Colorado (2023) says that a state may criminalize a statement if the defendant is reckless about the possibility of its being viewed as a threat, and not just if the defendant had the purpose of issuing a threat or knew that the statement would be viewed as a threat. Recklessness is defined as “consciously disregard[ing] a substantial [and unjustifiable] risk” that a human would see the statement and perceive it as threatening. Florida courts thus may allow punishment for reckless threats, without violating the First Amendment—though they could also stick with T.R.W. and require purpose or knowledge. See, e.g., C.R. v. Burke (Fla. App. 2026) (following T.R.W.‘s purpose/knowledge requirement, three years after Counterman was decided by the U.S. Supreme Court).

But in any event, it seems unlikely that the woman was consciously thinking “This might get read by a human, who might feel threatened, but I don’t care; I’ll just type it in.”  Rather, it seems much likelier that she was just assuming, as most of us do when using a chatbot, that we’re talking solely to the machine and that our input will never reach a human.  And if that’s so, then her statement just isn’t a threat punishable under section 836.10.

And of course a prosecution would have to prove the woman’s mental state beyond a reasonable doubt. So even if a jury thinks the woman might have been aware of the risk that this would be read by a human, that’s not enough: It would have to be convinced that there’s no reasonable possibility that she just assumed the text would go only to the chatbot. Such a finding of guilt beyond a reasonable doubt by the jury strikes me as quite unlikely, unless the accused had made some statement saying that she was hoping the statement would reach a human or at least knew that was going to happen (or maybe that she knew it was fairly likely).

The post Can a Chatbot Prompt Be a Punishable Threat? appeared first on Reason Magazine.