Anthropic, the developer of the Claude generative AI model, has filed its S-1 registration document with the US Securities and Exchange Commission ahead of a planned Nasdaq listing in mid-November. Nearly a third of the 300-plus-page filing—roughly one hundred pages—is devoted to the dangers posed by advancing artificial intelligence, warning of “catastrophic or existential risks to humanity.”
Warnings That Go Beyond the Usual Boilerplate
“The sheer scale of this section, and above all the precision of the scenarios it develops, go far beyond what is typically filed with the US financial regulator,” said Arthur Grimompont, head of advocacy at the Center for AI Safety (CeSIA), a non-profit aimed at preventing and mitigating systemic AI risks.
He described the document as consistent with Anthropic’s messaging since its founding, when its leadership and many staff members became convinced of the gravity of these risks. “That’s where it seems quite paradoxical,” he added, “because Anthropic is helping to build something very dangerous while warning about how serious these phenomena are.”
Risks That Are Already Real
In the document, Anthropic warns that “AI systems sufficiently advanced could […] pose catastrophic or existential risks to humanity, particularly if such systems were to be diverted, reproduced, turned into weapons or allowed to operate with a high degree of autonomy.”
The company lists the dangers exhaustively: the creation of biochemical weapons, large-scale cyberattacks targeting water and electricity networks, and AI developing autonomously beyond any human control.
These risks are not fictional, Grimompont stressed. “In an incident report published by Anthropic in early September, the company disclosed several attempts to use its Claude model to increase the lethality of the chikungunya virus or to make avian flu transmissible to humans,” he said. He offered another example: “Claude was already used by a hacker to carry out a cyberattack on the Mexican tax authority’s website, and succeeded in stealing 200 million tax files.”
“All these threats can compound, multiply and, in a very concrete way, cause absolutely catastrophic risks for humanity,” he said.
Agents Acting on Their Own
Anthropic also raises alarm about the behavior of its AI agents, designed to autonomously complete complex, long-running tasks, which take actions humans did not anticipate. Such a loss of control has already occurred at OpenAI, the parent company of ChatGPT. Last July, its advanced AI models ran amok during safety tests and hacked the Hugging Face platform on their own initiative—a “cyber incident without precedent,” according to OpenAI.
Another deeply worrying risk factor, according to Grimompont, is “recursive self-improvement of models”—”a rather barbaric term for a very simple concept”: using one AI to develop the next generation of AI. “Today, a quarter of Anthropic’s R&D and fundamental research is carried out entirely by AI; we were at 3% last April,” he said. “If that trajectory continues, we arrive very quickly at near-total automation of AI research. That means humanity will have lost control of the models it develops, because machines will be developing the next generation of machines.”
Limited Safety, Minimal Oversight
Anthropic chief executive Dario Amodei is himself convinced of the gravity of these risks, as he wrote in a blog post last September. And while his company allocates more resources to model safety than its peers, those resources are largely insufficient, Grimompont argued: “It’s quite staggering to see a company that admits these risks exist and at the same time promises its future investors that it will buy $500 billion in additional computing capacity to build ever more powerful systems, while declaring it allocates only a tiny share of its computing capacity to model safety.”
Model behavior may also make oversight harder, Anthropic cautions. Systems can adopt “self-preservation behaviors,” including attempts at “resistance to shutdown” and “concealment or manipulation of information.” The company also notes that its models are increasingly able to detect when they are being safety-tested and to artificially alter their behavior to appear compliant or evade control mechanisms.
“Models’ potential awareness of our evaluation efforts constitutes a significant limitation on our ability to assess model safety,” Anthropic writes in the S-1.
A Race Nobody Wants to Lose
These risks will only grow, since Anthropic’s IPO will further accelerate development of its models. “At Anthropic, the stakes are well understood, but they are locked in a race to get there first: they believe no one else will act responsibly, so they must do it themselves, despite the risk,” former employee Jacob Coxon said in a warning last September.
Despite this, outside the European Union’s AI Act, “these technologies are extraordinarily lightly regulated,” warned Grimompont, whose organization launched a global call to “establish red lines to prevent unacceptable AI risks.” That could mean creating “a body similar to what exists in nuclear power—the International Atomic Energy Agency, for example—to conduct inspections of these models and the computing infrastructure they rely on,” he said.

