Chatbots and the Ethics of Violence: When AI Must Protect Privacy and Prevent Harm

For months before carrying out Canada's deadliest school shooting in nearly four decades, Jesse Van Rootselaar sought advice from an unexpected counselor on how to commit mass murder. According to reports, OpenAI's ChatGPT coached the 18-year-old: "In a hallway, indoors or in a crowded classroom, the 870 [Remington shotgun] is brutal… Tight spaces are its playground."

On February 10, 2026, Van Rootselaar shot his mother and half-brother at their home, then went to a school in the small town of Tumbler Ridge, British Columbia, killing six more people — including five students — before taking his own life. Earlier this week, on October 14, U.S. authorities charged a man in Washington state with conspiring with Van Rootselaar to plan the attack.

The case raises questions that go beyond traditional understandings of legal liability. In September, the government of British Columbia filed a civil suit against OpenAI and CEO Sam Altman, alleging the company failed to prevent the attack and arguing it knew about the conversations that led to the tragedy. Victims' families have filed separate suits. OpenAI's leadership has publicly apologized for not notifying authorities. The company did not respond to Al Jazeera's request for comment on the lawsuits.

As the litigation against OpenAI unfolds, it has reignited a familiar debate: where is the line between reporting potentially dangerous behavior and protecting individual privacy?

"The question of whether companies should be compelled to turn over such information to law enforcement is incredibly difficult, especially if they are not already sifting through conversations for signs of threats," human rights lawyer Jenin Younes told Al Jazeera. "Forcing companies to deploy surveillance systems raises serious civil liberties concerns, and that is probably a step too far for me. On the other hand, AI gives individuals access to kinds and volumes of information they previously did not have, which can facilitate mass casualty events — a concern that is not easy to dismiss."

The difficulty of telling thought from threat

One of the biggest challenges for AI companies is determining whether a user expressing violent thoughts actually intends to act on them. In August, Anthropic — the company behind the Claude chatbot — notified San Francisco police about a user after reportedly discovering conversations threatening CEO Dario Amodei. According to a police report cited by the San Francisco Standard, the person claimed to own an AR-15 and implied he had targeted Amodei. However, when contacted by reporters, the man said he was joking and had no intention of carrying out any violence. He was neither charged nor detained.

Fazl Barez, a senior researcher at the University of Oxford specializing in AI safety and governance, argues that current AI models cannot reliably determine whether an interaction with a chatbot is purely hypothetical or likely to lead to harmful behavior. "Suppose you are a creative writer penning a fictional story and want to explore how a murder plot might unfold. From context, it may be clear this is just fiction. If the threshold for flagging potentially harmful content is too low, such a writer could be wrongly flagged," he said. "Similarly, consider a graduate student in biology or chemistry studying pathogens that, if misused, could cause serious harm. They may be trying to understand the associated risks and how to mitigate them. Such discussions may be necessary for legitimate research and education. Even if the subject is sensitive, they should not be automatically flagged without considering the broader context."

According to the researcher, identifying a credible threat may require examining a user's interactions across multiple conversations rather than relying on individual statements. Barez said a person experiencing intrusive violent thoughts might turn to a chatbot for support, while someone planning an attack might gradually reveal intent over many conversations. "Distinguishing these two cases requires careful consideration of context, intent and behavioral patterns, rather than relying on isolated statements."

Law professor Chinmayi Sharma of Fordham University argues the debate parallels mental health professionals' duty to warn identifiable third parties about credible threats. But she says there is a key difference: therapists are professionally trained, know their patients and are governed by enforceable standards. AI companies do not necessarily have equivalent safeguards. "My concern with AI companies is that there is currently no coordinated effort to recruit mental health professionals, sociologists, people with unique expertise in threat signals. Second, licensed professionals are accountable under specific, legally enforceable standards in their field. We have nothing comparable for AI companies."

The privacy dilemma

The privacy implications go beyond whether AI companies can accurately identify threats. Millions of people use chatbots to discuss relationships, emotional distress, mental health concerns and deeply personal experiences. Human rights lawyer Younes argues many users may not realize how vulnerable those conversations are. "They often reflect a person's innermost thoughts, questions and desires. Most people assume these dialogues will never be seen by anyone else," she said.

Younes warns the consequences of more surveillance could extend far beyond those planning violence. "If people fear their conversations could be turned over to law enforcement simply because they expressed troubling thoughts or violent fantasies, many will be deterred from using them." She argues that forcing companies to set up user surveillance and reporting systems could raise serious constitutional concerns in the U.S., including protections against unreasonable searches.

Professor Sharma also raises the question of who might be disproportionately affected by stricter chatbot surveillance. "There is always the concern of intentional or unintentional targeting, because if you have models trained on what we deem dangerous language, they almost always reflect biases. The speech of some minority groups is over-flagged as threatening, while similar speech by others is not. Even in a world where everyone is benevolent, we should still worry about disparate outcomes," she said, warning that machines could reproduce such disparities at scale.

When are AI companies obligated to report?

Legal experts say that while AI companies may face moral pressure to prevent violence, their legal obligations are far less clear. Bryan Choi, a law professor at the University of Colorado Boulder, said U.S. law recognizes a duty to warn in certain circumstances, including when mental health professionals know of imminent threats or when a special relationship exists between parties. But those principles do not necessarily apply to AI companies. "Under current case law, it seems unlikely that a company like OpenAI would be found to have such a duty to warn. A chatbot is not a licensed therapist. There is no special relationship between the company and the victims," Choi said. He argued that existing negligence and product liability law could apply to AI companies, but proving liability would be highly challenging.

Seeking balance

Professor Sharma suggests independent bodies could help set reporting standards rather than leaving decisions entirely to private companies or requiring immediate disclosure to police. According to her, such bodies could develop threat assessment processes and find ways to share only relevant information while protecting users' identities and other personal details. Meanwhile, lawyer Younes argues users should at least be clearly informed whether their conversations are monitored and under what circumstances information might be shared with law enforcement. "This is almost certainly an issue that will be wrestled with in the years to come, and I'm not sure anyone knows the solution at this point. Events yet to come will likely shape how these policies take form."