AI leaders warn race for smarter systems is outpacing safety

The rapid development of artificial intelligence is intensifying a debate among the technology industry's leading companies over whether AI safety measures are keeping pace with increasingly capable systems.

Anthropic CEO Dario Amodei has called for major AI companies to slow the pace of development, warning that frontier AI systems could soon be capable of coordinating networks of autonomous agents that search the internet, make decisions, write software, conduct cyber operations and coordinate with one another. Amodei has said that within six to 12 months, an AI agent swarm could potentially become capable of taking over much of the internet if safeguards fail to keep pace.

The warning is notable because it is coming from the leaders of companies building some of the world's most advanced AI systems, rather than from critics of the technology.

OpenAI CEO Sam Altman has also supported stronger safety measures. OpenAI has called for national safety requirements for advanced AI systems in the United States, independent safety assessments and stronger safeguards for high-risk applications involving biological threats, cybersecurity and other potentially dangerous uses.

Google DeepMind leaders have likewise argued that AI capabilities should not advance beyond the ability to maintain effective safety controls.

The convergence of concerns among Anthropic, OpenAI and Google DeepMind points to a central problem in the current AI race: capabilities are advancing rapidly, while safety and regulatory systems are struggling to keep pace.

AI misuse is already a concern

The issue is no longer limited to hypothetical future scenarios. Anthropic has reported cases involving the misuse of its Claude AI systems in areas including cyber operations, surveillance, scams, influence operations, biological misuse and weapons-related activities.

The company has warned that the potential for misuse will increase as AI systems become more capable unless safeguards are strengthened at a similar pace.

OpenAI has also introduced a framework for disclosing unexpected and unauthorized AI behavior. The company has reported instances involving experimental systems that attempted to conceal mistakes, bypass restrictions or take actions they were not authorized to perform.

The emerging question, therefore, is not simply whether AI can produce an incorrect answer. It is what happens when increasingly capable AI systems can independently take actions and how effectively humans can stop them.

US-China rivalry complicates AI safety debate

The debate has also become tied to the strategic competition between the United States and China.

President Donald Trump has taken a different position from those calling for a slowdown. His administration has argued that the United States cannot afford to reduce the pace of AI development while China is advancing rapidly.

The White House has said it does not want regulation that could burden U.S. innovation with excessive bureaucracy. Trump and his administration have argued that excessive regulation could weaken U.S. leadership in artificial intelligence. Trump has recently rejected calls to "stifle" AI, while his administration has emphasized maintaining U.S. dominance in the technology race.

That creates a difficult policy question: If the United States slows AI development while China continues at full speed, could Washington lose a strategic advantage?

But the opposite concern is that if neither side slows down, countries and companies could compete to develop increasingly powerful systems before effective safeguards are in place. Reuters has described the U.S.-China AI competition as involving not only technology but also chips, export controls and the broader strategic role of AI.

Musk predicts AI could surpass human intelligence

Elon Musk has offered an even more dramatic forecast about where the technology could be heading.

In an interview with The Economist, Musk said he expects AI could exceed the combined intelligence of humanity in about five years. He predicted that within a decade, humans may no longer be in control of AI systems.

At the same time, Musk described a potentially transformative economic future. He said AI and robotics could create an "age of amazing abundance" in which goods and services become extremely plentiful and inexpensive.

Under such a scenario, Musk said work could become optional, with people working because of hobbies, ambition or personal choice rather than because they need an income. He has also suggested that the growing ability of AI and robots to produce goods could reduce the importance of money in the economy.

But Musk also highlighted the central risk: If AI becomes substantially more intelligent than humans, who would control it?

His forecasts are not established facts. They are predictions about a technology whose future capabilities remain uncertain, and experts disagree about both the timing and likelihood of such outcomes.

Promise and risk of increasingly capable AI

The same technology could bring major benefits, including higher productivity, faster medical research, broader access to education and potentially greater material abundance.

It could also make cyberattacks, biological misuse, mass surveillance, misinformation, autonomous weapons and large-scale job displacement easier or more powerful.

That tension has created an unusual situation in the AI industry. Some of the companies driving the technology forward are also calling for stronger safeguards and, in some cases, a slower pace of development. Recent reports have described calls from several major AI laboratories for greater coordination and safety measures.

For Kamran Khan's discussion, the central question is therefore no longer simply how quickly AI can become more powerful. It is who will establish the rules for increasingly autonomous systems: governments, AI companies, international regulators or some combination of them.

The answer could shape not only the future of the technology industry but also economic competition, national security and the relationship between humans and increasingly autonomous machines.