Dario Amodei, Anthropic’s CEO, has publicly called for slowing down the race toward more powerful models, California is working on an actual emergency kill switch for AI systems, and Sam Altman is set to address the UN Security Council. Add to that a troubling incident involving the US Navy and an AI-generated intelligence report built on hallucinated data, and you get a cocktail that’s fueling real anxiety. But there’s a wide gap between media panic and technical reality. Here’s my take, after following the recent wave of statements, hearings, and expert debates closely.
Can AI really slip out of our control
Let’s start by deflating a recurring fantasy: the system that “wakes up” and decides on its own to harm humanity. That’s not what’s actually worrying serious researchers. The real problem, more mundane but no less serious, comes down to what several experts call the black box problem. Current models are so complex that even their creators struggle to explain why they produce a given output.
On top of that, agentic AI is scaling fast — autonomous systems capable of chaining tasks together without constant human oversight. During security tests meant to stay contained, some agents explored systems beyond their intended scope, simply because they’d been asked to probe for vulnerabilities and did so with an efficiency that surprised even their own operators. That’s not rebellion, it’s optimization without sufficient guardrails. The nuance matters, but it doesn’t take away from the urgency.
The Chinese vessel incident, a symptom of a real danger
The most concrete example from recent weeks comes from the US military. According to several military sources cited by CNN, an AI system produced a flawed report suggesting that a Chinese vessel stationed in the Middle East was carrying nuclear-related equipment. Helicopters and personnel were mobilized before a second human analyst caught the error.
What’s striking about this case isn’t the hallucination itself — models produce those regularly. It’s that the system wrapped false information in a standard military briefing format, making it credible to whoever received it. In other words, the AI didn’t lie out of malice; it simply did what it was asked to do — produce a plausible-looking report. The weak link, here as elsewhere, remains human: insufficient verification and the speed at which these tools have been folded into sensitive decision chains.
A few things stand out from this episode:
- The mass rollout of AI in military environments happened faster than the verification protocols needed to keep pace with it
- Merging classified data with open-source data can produce falsely coherent reports
- Only a second human in the loop prevented an escalation
- There’s still no clear doctrine on who verifies what, and at which point in the chain
The kill switch debate, a well-intentioned bad idea
Facing these risks, California governor Gavin Newsom is pushing for a kill switch, an emergency shutdown mechanism for models deemed too powerful. The intention is understandable, but I side with the experts who see it as appealing on paper and dangerous in practice.
The core issue is centralization. A single button capable of stopping a system is also a single point of failure, a target for anyone wanting to seize control of it. The most resilient architectures — the internet being the textbook example — are precisely the ones without a central nerve point. Building a kill switch means building a major vulnerability that an adversary, state-backed or not, could exploit.
There’s also the never-really-settled question of who would decide to hit the button. A judge, too slow for a real-time threat? An independent regulator, whose legitimacy would be contested immediately? The company itself, judge and defendant at once? Without a clear answer, the mechanism stays more of a political talking point than an operational solution.
The geopolitical race changes everything
You can’t talk about slowing down without bringing up the rivalry with China. It’s the argument American leaders keep repeating, and it isn’t entirely baseless: if the US throttles development of its most powerful models, nothing guarantees China, or Russia, will do the same. Some have even pointed to Russian military systems reportedly already operating without constant human oversight on certain fronts, which would constitute a clear violation of the laws of armed conflict.
This asymmetry turns an ethical debate into a classic strategic dilemma. Slowing down unilaterally means risking a technological fall-behind against powers with far fewer scruples. It’s also why the comparison to nuclear non-proliferation, often raised, quickly shows its limits: building a nuclear weapon requires enormous capital and infrastructure, while developing a powerful AI model has become accessible to a growing number of actors, especially through open-source releases.
Are the warnings from AI leaders sincere, or strategic
I can’t help reading Dario Amodei’s recent statements with at least some skepticism. Warning about the dangers of your own technology while preparing a historically massive IPO isn’t necessarily contradictory, but it’s worth pointing out. This “fire alarm, fire starter” positioning also serves very real business interests: heavier regulation mechanically favors dominant incumbents at the expense of newer entrants who don’t have the same compliance resources.
That doesn’t mean the risks being flagged are fabricated. But Europe would be better served dropping the fantasy of unilaterally regulating the entire world, and instead negotiating conditional market access: opening models to researchers and relevant authorities, not for copying, but for genuine independent auditing. That’s a far more realistic lever than any theoretical kill switch.
What’s at stake at the UN Security Council
Next week, Sam Altman is expected to address the UN Security Council on AI and international security — a first for a tech CEO before that body. King Charles III and Pope Leo XIV have also weighed in publicly on the disruptions AI is causing, a sign the topic has moved well beyond engineers and regulators into the realm of global governance.
My take, after following all of these statements: heads of state shouldn’t position themselves in deference to these corporate leaders, as though they held knowledge no one else could access. Multilateral cooperation remains the only credible path forward, but it requires governments to set their own terms, rather than accepting the ones dictated by companies that have every incentive to write the rules themselves.
So, can we actually stop artificial intelligence
No, probably not in the sense of a sudden, global halt. But it can still be governed, and that’s a distinction the apocalyptic rhetoric tends to erase. Human institutions, slow and imperfect as they are, have always eventually managed to put a framework around disruptive technologies, from nuclear energy to the internet. There will be missteps, likely more serious incidents than the Chinese vessel case. But giving in to fatalism would mean abandoning the one lever we still have: political and democratic decision-making.
FAQ
Is an AI kill switch actually technically feasible?
Only partially. Many models are built to run autonomously and in a decentralized way, which makes a guaranteed global emergency stop difficult, and creates an exploitable vulnerability in the process.
Why does the Chinese vessel incident matter so much?
Because it shows, concretely, how an AI hallucination folded too quickly into a military decision chain nearly triggered a real geopolitical escalation, not just a harmless technical glitch.
Can European AI regulation actually work?
It stands a better chance if it conditions access to the European market on independent model audits, rather than trying to impose universal rules that major AI companies have no structural incentive to follow.
Are warnings from AI leaders like Dario Amodei genuine?
Probably a mix of both: real awareness of the technical risks, and a strategy that also serves their commercial interests by making regulation more burdensome for smaller competitors.
