Anthropic, the developer of the AI model Claude and the maker of Mythos, is calling for a worldwide temporary pause on artificial intelligence development. The company, valued between $965 billion and $1 trillion, warned that cutting-edge models are beginning to show signs that they could escape human control. Anthropic is urging the world's top AI laboratories to consider slowing down to allow society to deal with the immense implications of the technology.
The warning centers on the concept of recursive self-improvement, where AI systems become capable of making better and more powerful versions of themselves without human intervention. Anthropic argues that these rapid advances could soon allow AI to improve faster than humans can manage the consequences. This capability is viewed by safety researchers as a key step toward superintelligence, which could unleash widespread consequences on humanity.
To address these risks, Anthropic suggests the creation of a coordinated and verifiable industrywide mechanism to pause development. Such an agreement would require major AI companies across multiple countries, most notably the United States and China, to stop development simultaneously under verifiable rules. The company has also floated the idea of convening policymakers to discuss the dangers of advanced AI.
This call for a pause comes as Anthropic heads toward an initial public offering of stock. While the company argues that a pause would likely be a good thing, some experts view the announcement and the proposal to convene policymakers as a marketing ploy.