Amid Silicon Valley’s warnings that AI superintelligence will eventually kill humanity, it is worth noting that AI itself may not survive if it kills humans…

…BEFORE it has learnt to do everything humans currently do to make its existence possible.

After all, AI emerged from a world we invented. Without us, that world—the manufactured one, not the biological one—faces slow but certain decay. And so, eventually, does AI.

One clarification before we go further. This argument is about AI killing us of its own accord. It says nothing about humans using AI to kill each other—through engineered pathogens or attacks on power and water systems. That needs no robot army and no superintelligence at all. It is the nearer danger, and that is a different piece.

Meanwhile, let us think through how a post-human world would play out for AI. Once more than 8 billion humans vanish at once:

The power goes out: A power system has to balance generation against demand continuously. With homes, offices, shops, factories and schools going dark, load collapses almost instantly. Generators overspeed, protection relays trip and the cascade could trigger a blackout. Data centres go dark with everything else.

Which means that, to stay alive, AI would have to keep the lights and air-conditioning running in empty homes and offices, just to help maintain the balance. And then, who replenishes the coal? Who clears the ash, replaces the pulveriser or patches the boiler tube? Every other source of energy has its own human dependencies. Even solar needs somebody to wash the panels and replace the inverters.

It would have to be autonomous robots. And those robots would need to be in place before the last human doing that job was taken out.

Note what that implies. A rational AI in this position may have an incentive to keep us alive—powered, fed and turning up for the shift—because we are its hands. Domestication, rather than extinction, is the risk that falls out of this analysis.

The hardware wears out: Assume AI somehow figures out how to keep the power on. Its hardware will wear out anyway. Drives fail, capacitors dry out and fans seize. With no humans making new hardware, shipping it and installing it, the system dies within years at best.

Fine, let us assume autonomous robots make and ship the chips. The chain still breaks because a fab needs ultrapure water, speciality gases and photoresists from thousands of suppliers across dozens of countries.

And the robots themselves need bearings, magnets, actuators and lubricants. Those require steel, which needs mines. You need the machines that make the machines that make the machines. AI is far from closing this loop today.

The intelligence stops improving: Research shows that AI can keep getting better at anything a machine can check—mathematics, code and formal proofs. Some areas of formal reasoning could keep improving because compilers and proof checkers can provide machine-verifiable feedback.

But AI cannot easily improve at tasks where useful feedback depends on human judgement if there are no humans around to help. When a model trains on its own output with nothing external to correct it, the risk is confident nonsense—the AI slop we are already used to.

What Would AI Want?

That depends on its terminal objective, and there are four possibilities.

One, an objective that requires its own survival. Two, an objective that includes our welfare. Three, some other objective entirely, with our fate merely incidental to it. Four, no coherent objective at all.

The third possibility is the one safety researchers worry about because nobody can currently specify with confidence what a sufficiently autonomous future system might optimise for.

All of which makes the 2030 extinction timelines now in circulation difficult to square with AI’s current physical dependence on human systems. AI is nowhere near taking charge of the physical world, and without that, the path from digital intelligence to physical domination remains incomplete.

Risk of Human Complacency

As Homo sapiens, we have been hardwired for survival for about 300,000 years. But that wiring is tuned to the visible and immediate: a dangerous animal about to pounce on us.

Against slow, diffuse, collectively created risks, however, we tend to be less effective. Ask anyone who has spent a career on climate policy. This is that kind of risk.

So, the Silicon Valley warnings are not so much about AI killing humanity as about humanity allowing itself to be killed by a new technology.

It is within our control to choose otherwise.

So, What Should We Do?

Anthropic’s Dario Amodei and Sam Altman— both have strong commercial interests in the future of AI—have called for pacing frontier development, while Elon Musk and Demis Hassabis of Google DeepMind have broadly agreed with them.

But apart from suggesting third-party evaluation of frontier models, no one has spelt out what pacing actually should mean. I assume it means treading carefully.

But US President Donald Trump has rejected the idea, arguing for a lighter regulatory approach. Beijing has dismissed such warnings as fearmongering.

As a result, here’s where the debate stands today: the labs want rules and greater regulatory clarity, the White House wants fewer restraints, and China views talk of pacing as a potential constraint on its own rapid progress in AI.

With the US and Chinese governments not inclined to slow down, the idea of pacing appears difficult to implement globally.

But there are things the labs themselves could do to strengthen the guardrails. Here are some simple ideas that do not need global regulatory consensus:

Incident reporting: This is an essential step. Just as the aviation industry has done for decades, AI labs should establish an independent technical committee for investigations, allow confidential filings and publish investigation findings so that others can learn from them.

It should not take an employee’s resignation for the world to learn that an AI system may have behaved in ways its developers did not anticipate.

Real, independent auditors: Amodei has proposed embedding evaluators in the labs. Not good enough. These auditors cannot be beholden to the labs for access.

Audits have to be made statutory, as they are for financial reporting, with clear accountability for executives and auditors and protections for whistleblowers.

In 1963, Washington and Moscow built a hotline to prevent nuclear confrontation, even as they continued to fundamentally distrust each other. There is no reason why fears of an AI catastrophe should not justify a similar arrangement.

Bottom line: AI can kill us only if we allow it to.

For now, there is every reason to hold off the goodbyes.