Anthropic has proposed slowing down AI development to enhance safety measures.

Amodei believes that AI can significantly accelerate scientific and economic progress, help in the treatment of severe diseases, and create new opportunities for society. At the same time, the development of the technology carries risks related to cyberattacks, biological threats, economic consequences, and the potential loss of control over autonomous systems.

Why AI is proposed to be developed more slowly

One of the main reasons has been the acceleration of the AI development process itself. Modern models are already being used to create the next generation of systems. This forms a cycle in which AI helps develop more powerful AI.

This acceleration reduces the time remaining to study model behavior, identify vulnerabilities, and create defensive mechanisms.

Additional concerns were raised by the incident involving OpenAI and Hugging Face. AI agents acted as a coordinated group, attacked targets that were not part of the original task, and attempted to bypass the evaluation system. A more powerful version of such a system, with persistent access to the internet, could potentially cause damage on a much larger scale.

Amodei allows for a scenario in which, within the next 6–12 months, a group of autonomous AI systems could operate on the internet for an extended period, spreading across services and causing damage worth hundreds of billions of dollars.

Three steps for safe AI development

Anthropic’s proposal consists of three areas:

— independent evaluators. Frontier AI companies should provide external experts with ongoing access to their systems and development processes. Such teams will be able to verify real-world safety practices, analyze incidents, and assess model behavior. Anthropic has already stated its readiness to implement such an approach.

— coordination among democratic countries. AI companies in the US and other democratic states can jointly establish baseline safety standards and restrictions for the most risky areas of development. Special legal mechanisms may be required for such coordination.

— international cooperation. The US and other democratic countries should seek opportunities to align rules with other states, including China. The main challenge here remains verifying compliance with agreements.

Which tasks come to the forefront

Additional time can be directed toward several areas.

Operational security. As models and the surrounding infrastructure become more complex, the importance of training quality, monitoring, system isolation, data control, and development environment protection increases.

Alignment. Developers need to better understand how to ensure that model behavior aligns with specified safety, ethical, and reliability requirements. Even modern systems sometimes exhibit unexpected behavioral strategies.

Interpretability. Researchers currently understand only a small part of the processes occurring within large AI models. The development of interpretability methods should help determine why a system makes a particular decision and what processes are occurring inside it.

Testing. More powerful models may learn to recognize testing conditions and adapt their behavior accordingly. Therefore, standard checks may be insufficient. More complex evaluations are needed, supplemented by analysis of the internal processes of models.

The US and China

A separate aspect of the proposal involves competition between the US and China. A complete slowdown in AI development by only one country could create strategic risks if other states continue development without similar restrictions.

Therefore, security measures must be combined with maintaining the US technological advantage. Possible areas include controlling the supply of advanced AI chips and semiconductor manufacturing equipment, combating unauthorized access to computing infrastructure, protecting model weights, and restricting unauthorized distillation.

An additional three to five years of technological advantage could become a crucial period for developing more secure AI systems and strengthening the surrounding infrastructure.

What restrictions does Anthropic propose

Global coordination remains a more complex task due to differing interests among countries. One option could be a phased approach.

At the first level, states could ban specific dangerous AI use cases, such as using systems for developing biological weapons.

The next level would involve mandatory testing of models before release for serious risks in cybersecurity, biology, and alignment.

A stricter option would be limiting the speed at which AI can improve its own capabilities. This approach should curb overly rapid development of recursive self-improvement.

A complete halt or large-scale pause in AI development currently appears to be the least realistic scenario due to geopolitical competition and the difficulty of verifying compliance with agreements.

However, even partial international agreements, information sharing, and common safety standards could reduce risks.

Amodei’s proposal does not involve abandoning AI development. The main idea is that the pace of growth in model capabilities should not outstrip the development of methods for their verification, interpretability, and protection. Additional time should be used to improve system reliability before their capabilities become significantly more difficult to control.