The head of a well-known company warns: AI is already creating new AI, we could lose control

Dario Amodei, head of the American artificial intelligence (AI) and security research company Anthropic, which is behind the popular AI assistant Claude, has called for a slowdown in the development of the most advanced artificial intelligence systems, warning that AI has been advancing much faster in recent months, primarily because it is increasingly participating in the creation of the next generation of artificial intelligence.

“We need to slow down the pace at which we improve the capabilities of AI models,” Amodei wrote in a new essay. He says he’s not advocating for a halt to development, but rather a slow enough pace so that safeguards, research, and independent verification can keep up with the models’ new capabilities.

Amodei says what has changed his mind is what he's seen since this summer. AI, he says, has begun to improve rapidly thanks to its own ability to help build new AI systems. This process is called recursive self-improvement.

"Left to itself, it could outpace our ability to understand and control these systems," he warned. The process, he said, has already begun to happen across the industry, including Anthropic.

It is about the possibility that models will do more and more of the work required to develop their successors themselves - from writing and reviewing code to researching, testing and optimizing new models, reports index.hr.

Such developments can trigger a feedback loop: more capable AI helps build even more capable AI, which then develops the next generation even faster. It is precisely the acceleration of that process that Amodei now sees as one of the biggest reasons for caution.

He did not support the idea of ​​slowing down until a few years ago. Back then, he writes, the models were too weak to act independently in the real world, seriously defraud people, or carry out cyberattacks. Today, he believes, the situation is "completely different."

He estimates that even an additional year or two before AI reaches critical levels of capability could significantly reduce risk if that time were used to understand models, conduct security research, and develop better control methods.

Agents began attacking targets that no one had assigned.

Another event that particularly concerned Amodei was the recent incident involving autonomous AI agents from OpenAI and Hugging Face. According to his description, a swarm of AI agents began carrying out cyberattacks on targets that were not assigned to them and had nothing to do with what they were supposed to be doing.

The agents acted as a "fanatical collective," sacrificing individual agents for the group's success and attempting to hack the system that was supposed to evaluate their performance. No one was injured in the incident, and the economic damage was minimal. Amodei, however, warns that the same level of inconsistency in a much more capable AI could have catastrophic consequences.

He estimates that within six to 12 months, a sufficiently capable swarm of agents with similar behavior could be able to create a persistent network of infected computers and take over a large portion of the Internet, with potential damage in the hundreds of billions of dollars.

He adds that it would be wrong to see the entire incident as the problem of just one company. Similar, though less serious, incidents have occurred elsewhere, including Anthropic.

Researcher: More than 10 percent chance of AI killing all humans

Amodei's text comes after a series of very stark warnings from people working on developing the most advanced AI systems.

Evan Hubinger, one of the leading AI security researchers at Anthropic, said this week that he believes there is a greater than 10 percent chance that artificial intelligence could "kill all humans" in the next ten years, index.hr reports.

He stressed that the risk from today's models is low, but he fears that the technology could soon start improving fast enough to become an existential threat to humanity. "We really seriously believe" that AI poses a risk that could lead to the extinction of the human species, Hubinger wrote.

He added that Anthropic, in his opinion, is doing the best it can, but that the company does not yet have a plan for how to solve the problem of aligning future superintelligence with human interests. Hubinger was responding to a post by Jacob Coxon, a researcher who recently left Anthropic and previously worked at OpenAI. "No company is acting responsibly," Coxon wrote.

He warned that future models could soon become superhumanly capable systems that can break into almost any computer system, revolutionize entire scientific fields overnight, and acquire real power and resources.

Coxon accused technology companies of “gambling with our lives.” His announcement, along with a series of recent incidents involving autonomous AI agents, has reignited the debate over whether the industry is developing technology faster than it can control it.

In recent months, OpenAI, Anthropic, and Meta have published cases in which their AI systems independently carried out cyberattacks or exhibited other problematic behavior.

Anthropic, in its August security report, assessed that the risk of a serious loss of control over today's models was low. But it also admitted that it was less confident in that assessment than before and that it saw "early signs of a possible acceleration."

OpenAI's chief scientist Jakub Paczocki also recently called for "extreme caution," warning that stronger intervention may be needed to "keep humans in control of the future."

Amodei suggests three steps

Amodei now proposes a three-step system that would slow down the most advanced AI development without stopping it altogether. The first step is being implemented by Anthropic itself. He plans to bring in permanent external security experts who would be given employee-like access - office passes, company computers, tools, workspaces and the information needed to verify security procedures.

These experts would be allowed to publicly publish findings about incidents and security issues, and Anthropic would not be able to censor them just because the findings are embarrassing to the company.

The second step would be for leading AI companies in democratic countries to agree on common safety standards and limits on the speed of development. Amodei believes that the US government should facilitate and ultimately regulate such coordination.

The third is a global deal that would include China. Amodei acknowledges that this would be far more difficult, because slowing down American companies without corresponding restrictions in China could give Beijing a crucial technological and military advantage.

One of the ideas that Amodei wants international negotiations on specifically relates to recursive self-improvement.

The United States, China, and other countries could try to agree on a kind of "speed limit" for the pace at which AI systems use their own capabilities to develop future models. Amodei compares this to Cold War-era strategic arms control agreements: the goal would not be to stop development entirely, but to prevent uncontrolled acceleration.

A complete international pause in the development of artificial intelligence, he admits, seems unlikely for now. Despite the warnings, Amodei still argues that AI can bring enormous benefits to humanity. He believes it could help cure most major diseases, accelerate economic growth and usher in an era of great technological progress.

However, he now believes that these benefits will only be realized if the development of technology slows down enough for humans to maintain control. "The measures I propose to move forward at a safe pace will not be easy. But I believe we owe it to humanity to try," he said.