Anthropic calls for pause on AI development, warns of loss of control

Anthropic, the company behind the Claude chatbot, has proposed that leading AI labs find ways to coordinate a pause on developing advanced AI systems, warning that the technology is improving so fast that humans risk losing control.

In a blog post on Thursday, Anthropic said that as advanced AI increasingly performs tasks faster, “it would be good for the world to have the option to slow down or pause” its development. The company said its internal research institute plans to explore the issue with others and “take actions” to help build systems for reliable slowing or pausing, without giving details.

Anthropic’s rival OpenAI took a different approach in a report published on Wednesday, arguing that “democratic governments — not private companies acting alone — must ultimately determine the rules, safeguards, and accountability mechanisms.” “Our view is that decisions about the pace of AI innovation should not be left to any single lab, company, or special interest group,” OpenAI said.

Anthropic said AI models are becoming faster, with rapid increases in the speed of software tasks such as writing code. Based on current trends and with enough computing power, an AI system could design and develop its own successor, a process called “recursive self-improvement.”

Self-building AI would be a major technological milestone, bringing benefits in science, medicine and other fields, but “could also increase the risk of humans losing control of AI systems,” Anthropic said. Some tech figures have long warned about such a scenario.

Anthropic’s post followed another warning this week from researchers at the University of Toronto, who showed how AI tools could create a new type of AI “worm” that adapts its attack strategy as it spreads from device to device and takes over a wide computer network.

“I think it’s really important that people understand that it’s not just the biggest, most powerful language models that raise security concerns,” lead researcher Nicolas Papernot said in an interview.

The authors of the Anthropic post, co-founder Jack Clark and Marina Favaro, head of the research institute, said a pause would be used to allow “social structures and alignment research” to catch up with AI advances. “Alignment” is the industry term for ensuring technology matches human values and intentions.

The proposed coordination would allow advanced AI labs to verify that global rivals have actually paused or slowed their work, “and that a bad actor cannot exploit the coordinated slowdown to secretly race ahead.” The company said a global coordination mechanism is needed because, without it, slowing AI development could allow “the least cautious players” to catch up and put more pressure on companies and governments as they make tough choices on AI safety.

Concerns that advanced AI systems could escape human control and harm society have grown as technology becomes more capable. Anthropic’s own Mythos model shook industries, including banking and software, earlier this year with its ability to find flaws in existing code.

However, regulation is slow, especially in the US, home to most leading AI labs. An executive order from the Trump administration earlier this week put responsibility on labs, asking them to voluntarily submit their most powerful models for government cybersecurity checks before public release.

AI researchers have previously called for pauses, with limited success. Elon Musk, owner of the xAI lab, was among supporters of a 2023 effort by the non-profit Future of Life Institute to halt AI development for six months to establish safety safeguards.

Anthropic has long positioned itself as a safety-focused AI lab. Earlier this year, it refused to let the US military use its models for domestic surveillance and fully autonomous weapons, drawing backlash from the government, which placed the company on a national security blacklist to take effect by late 2026.

Anthropic’s post comes as it and ChatGPT-maker OpenAI race to sell shares to public markets in an IPO that could value Anthropic at nearly one trillion dollars.

Papernot notified Canadian cybersecurity agencies before publishing his report, showing how researchers built the worm in a lab using an “open-source” AI tool easily accessible and modifiable by software developers at low cost.

“In the past, cyber attackers would focus on very high-value targets,” he said: “Banking systems, hospitals, power grids, water treatment plants, schools.” Papernot agreed that more cooperation is needed among companies, government agencies and academic researchers to develop countermeasures as AI-powered attack tools accelerate the search for computer vulnerabilities.

“That old laptop in the basement you don’t check often may not seem like a high-value target, but it could be used as a steppingstone to attack higher-value targets,” he said. “Anything internet-connected is now at risk because the cost of executing these cyberattacks has become so low.”