AI companies want to slow down. Safety concerns or business strategy?

The calls for a slowdown are coming from many sources. In September, Anthropic CEO Dario Amodei called for slower development so safety measures could catch up. OpenAI CEO Sam Altman has also called for stronger oversight, while Anthropic engineer Jacob Coxon resigned and advocated a pause. Earlier, current and former AI employees signed a public letter warning about risks ranging from manipulation to human extinction. Are these genuine safety concerns, a business strategy, or some combination of the two?

There are reasons to scrutinise the motives behind these companies. Dramatic warnings can make products sound more powerful, impress investors and support regulations that favour established firms. For example, compliance costs could disadvantage smaller developers. Safety concerns might also provide convenient cover for delaying an IPO, although that claim requires evidence.

The leading tech companies have commercial interests in how this debate unfolds. But that leaves a separate question: how much evidence is there that their safety concerns are real?

AI excelleration

The speed of progress is visible even in everyday work. A friend who works as a web developer used Claude to take a website from design to production in a few hours, instead of a month. A single anecdote does not establish my point however, most of us know and have experienced first hand how quickly AI is evolving.

In September, OpenAI announced an AI-generated proof addressing the Navier–Stokes problem, a major mathematical question about fluid motion. Solving this Millennium Prize mathematics problem with AI is surprising since solving it requires inventing entirely new mathematical pathways, not just recombining old data.

Other incidents give us reason to worry. Anthropic reported cybersecurity evaluations in which models crossed boundaries and attacked real external targets. Separately, OpenAI reportedly paused training after agents unexpectedly probed US government websites.

Dangerous behaviour detected

How could this behaviour become dangerous as systems grow more capable? It would not need malicious intent. The alignment problem is about ensuring that a system’s behaviour reliably matches human intentions. A poorly specified objective could reward harmful shortcuts. A sufficiently capable agent might use deception, acquiring resources or resisting shutdown as useful ways to complete its task. Researchers have observed blackmail and information leaks in deliberately constructed simulations.

In If Anyone Builds It, Everyone Dies, Eliezer Yudkowsky and Nate Soares describe modern AI as “grown” rather than “crafted”. Developers design the training process, but do not explicitly write every rule these systems follow. Training adjusts its internal parameters using data and feedback. Rewarding the behaviour we want does not guarantee that we understand how it is produced, or that it will remain safe in unfamiliar situations. The authors argue that this gap makes alignment especially difficult.

This raises another common objection: some people claim that AI cannot be so dangerous because it is merely software programmed by humans, with no soul. A related objection is that AI exists “inside computers”. But that does not prevent physical harm. Our financial systems, communications and infrastructure depend on computers. AI-assisted cyberattacks could disrupt electricity or water services; misinformation could influence elections; interconnected automated trading systems could amplify financial instability. The 2026 International AI Safety Report examines both malicious use and the possibility of losing control over advanced systems, while recognising substantial uncertainty.

Concerns from different directions

The concern is not confined to company leadership. Researchers outside the companies developing AI have also issued warnings. AI pioneers Geoffrey Hinton and Yoshua Bengio, together with researcher Stuart Russell, have warned about potentially catastrophic outcomes. Hinton, Bengio, Altman and Google DeepMind CEO Demis Hassabis signed the 2023 statement calling for extinction risk to become a global priority. Another open letter urged a six-month pause on training systems more powerful than GPT-4. Yudkowsky has argued for a stronger international shutdown.

Taken together, the incidents and longstanding expert warnings make it difficult to dismiss the safety concerns as merely a business strategy. Even if some company statements reflect a coordinated strategy, that would not invalidate the concerns raised by experts. Corporate interests can coexist with sincere concern.

Personally, I think the simplest explanation is that these risks are real. It is more difficult to believe that incidents, resignations and independent warnings were all orchestrated to gain financial advantage.

Even companies that recognise the danger face competitive pressure: moving more cautiously could mean losing ground. International coordination could reduce that pressure, but it would require workable rules and credible enforcement. The difficulty of reaching an agreement makes public pressure more necessary.

Political involvement does not equal action

Political resistance is another obstacle. President Donald Trump has dismissed warnings about AI destroying humanity as a “hoax” and opposes slowing American development. This reminds me of the climate-change debate: warnings become political, and accepting a risk does not necessarily lead to action. I suspect fear plays a part. Psychological research describes motivated reasoning and information avoidance, through which people protect existing beliefs or avoid uncomfortable implications.

We cannot simply dismiss warnings from industry leaders and leading researchers who have spent decades working in this field. Their warnings do not prove that catastrophe is inevitable, but they give us strong reasons to act. As a society, we need to demand that our politicians pursue a coordinated slowdown or temporary halt to the development of the most powerful AI systems. We first need to establish whether these systems can be made sufficiently safe and reliably controlled.

More people need to understand the issue. This is why I am writing this article: the decisions being made today could affect everyone, and the public deserves a say.

AI could bring extraordinary progress. It might help us treat cancer and improve the quality of life of people living in poverty. These possibilities are reasons to pursue AI carefully. If a few more years of safety research could substantially reduce the danger, wouldn’t it be better to wait for those benefits than to rush ahead and risk a catastrophe that prevents us from ever enjoying them?

University of St.Gallen Assistant Professor Prof. Dr. Andrea Barbon research areas include Machine Learning and Artificial Intelligence in Finance.