Why Intelligent People Keep Moving Toward Outcomes They Know Are Dangerous

Jacob Coxon did something that none of us have done in a race. He stopped.

Coxon resigned from Anthropic and said, publicly, that the labs are racing toward a catastrophic outcome they already know is possible. His colleague Evan Hubinger quantified the probability of AI-caused human extinction above ten percent within a decade, adding that the industry does not yet have a plan for aligning superintelligence, and is not clearly on track to develop one. Both men remain on record and yet, the labs kept building.

And now, from social media to coffee tables we all have discussed this story as an AI cautionary tale, as a tech story, as a story of safety versus speed, a story of creatives versus suits. But there is another story here. And it's ours.

It is a story of how we can recognize the danger, explain it better than anyone else and still continue towards it.

Most of us assume that seeing a threat is the beginning of doing something about it. That assumption becomes unreliable when the threat is clear but agency is not.

When you can see the danger but cannot see a viable way to stop it, awareness becomes corrosive. Instead of action, it produces vigilance and vigilance is not the same thing as deliberation.

To monitor a threat is to pay attention to it. To respond to a threat is to change the conditions producing it. These are different operations, but most institutional safety languages collapse them into one.

Imagine a threat that never resolves. You monitor it, discuss it, build systems to detect it and even warn people about it. But the one action that would remove the threat is unavailable. Because that action is stopping, how can you stop when everyone is running? You will get crushed. The threat is in future, getting crushed is immediate. What will you choose?

In the case of the AI race, the immediate threat is the competitor, it will not stop even if you would. In any boardroom the immediate threat is almost always the competitor. It is concrete, measurable and present. The catastrophe is probabilistic. It belongs to a future nobody can see. So `if we stop, they won't` feels like a real danger, while this may eventually become catastrophic remains a possibility.

This is the situation the competitive architecture of the modern workplace is putting our mind into. We are asking it to remain alert to the danger while continuing to participate in the environment producing it.

It changes what the mind is optimised for. Under sustained threat, attention becomes more tightly captured by what is immediate, salient and actionable. Speed becomes more valuable. The future becomes harder to weigh against the present.

This makes perfect sense if a car is coming towards you. You do not need a twenty-year scenario analysis. You need to move.

The problem is what happens when an environment keeps sending the body the signal that a car is coming for ten years. It is what the state does to weighting. And that changes the moral equation.

Every laboratory can believe that it is the responsible actor. Every laboratory can believe the others are the greater risk. Every laboratory can therefore conclude that its own continuation is the safer choice.

If Anthropic slows down but another laboratory does not, the technology does not disappear. This has the structure of a prisoner's dilemma: restraint works only if restraint is sufficiently reciprocal. Unilateral restraint can look less like safety and more like surrender.

Nobody needs to be lying. Nobody needs to be greedy. Nobody even needs to be irrational. The system can make individually defensible decisions add up to a collectively dangerous outcome.

We have seen it happen before. In the Space-race, the U.S.  looked down upon the Soviet space program initially, only to be repeatedly humiliated when the Soviets beat them to almost every other major milestone. The Soviets were the first to put a satellite in orbit, the first to send a man into space, and the first to achieve a robotic soft-landing on the Moon. This string of defeats is exactly what triggered the frantic, high-pressure "rush" by the U.S. to claim the ultimate prize: a crewed lunar landing.

And then, on January 27, 1967, Virgil Grissom, Edward White and Roger Chaffee died in a fire during a preflight ground test of Apollo 1. NASA's investigation identified multiple contributing conditions: a 100% oxygen atmosphere under pressure, extensive combustible materials, vulnerable wiring and plumbing, inadequate escape provisions, and inadequate rescue preparations. Crucially, the review board found that the organisations responsible for planning, conducting and ensuring the safety of the test failed to identify the test as hazardous.

The people involved did not need to be reckless or indifferent to produce catastrophe. A system organised around an extraordinary goal accumulated assumptions, shortcuts and unresolved hazards until individually defensible decisions became collectively lethal.

It was not that space-race was inherently bad, but when an institution becomes psychologically and organisationally committed to an outcome, warnings can be absorbed as engineering problems rather than signals to reconsider the trajectory.

We all have seen it first hand, when a senior executive knows a strategy is failing but cannot be the first to abandon it. A company knows its culture is damaging but believes changing first will make it less competitive. An investor knows the position is wrong but selling alone crystallises the loss. A person knows a relationship has become destructive but staying postpones consequences that leaving makes immediate.

We call these failures of courage, discipline or judgment. Sometimes they are. And then, there are times when the problem is structural.

Remember the Lion Air 610 and Ethiopian Airlines 302 crashes? Both involved Boeing 737 MAX aircraft, and all 346 people aboard the two flights died. The NTSB's examination found problems in the assumptions used in the original safety assessments of MCAS and specifically highlighted a gap between how Boeing and the FAA assumed pilots would respond and how crews actually experienced the multiple alerts during the accidents.

Nobody had to wake up and decide: Let's build an aircraft that kills people. The catastrophe emerged from the interaction of decisions that each had their own institutional rationale.

Apollo and Boeing are different failures. But they reveal the same uncomfortable property of complex systems: catastrophe does not require a catastrophic decision. Knowing what should happen is not the same as being in a situation where doing it is possible.

And, what happens to the people who cannot make peace with the contradiction?

Psychology has long examined differences in how strongly people respond to their environments. The orchid vs dandelion theory says that some people are more sensitive to environmental conditions, benefiting more from supportive environments and being more affected by adverse ones.

What happens when that sensitivity meets an institution that continuously recreates the threat?

Perhaps some of the people who find the contradiction hardest to normalise leave first because they cannot make the environment psychologically coherent. The people who remain may, over time, become disproportionately comfortable with a contradiction that once disturbed others.

This is not an established law of organisational behaviour. But it is a possibility worth examining. Because if it is even partly true, an institution can lose part of its capacity to hear its own warnings without ever deciding to do so.

The people most disturbed by the warning leave. The warning becomes easier for everyone else to live with. The institution did not decide it. Its environment made that choice for it.

Dhritarashtra knows enough to understand where the conflict is heading. He hears the warnings. He knows the consequences are becoming harder to contain. But recognition does not become intervention.

The obstacle is not simply ignorance. It is the structure of loyalty, power, family, position and consequence surrounding the decision. Stopping has become psychologically and politically more expensive than continuing.

The AI problem is more modern, but the human mechanism is older. Everyone can see the danger, everyone can explain the danger, and the structure still makes stopping individually irrational.

No Krishna can solve that with a better argument. Because the missing ingredient is not information. It is coordination.

Coxon wants a coordinated restraint. But, coordinated restraint is a long-horizon, deliberative decision. And, the physiological state the threat produces is incompatible with long-horizon, deliberative decisions. So the decision he is asking for is ruled out in advance. And, it is not by opposition, but by the state the situation creates.

The natural conclusion to a story about a brave resignation is "more people should do what Coxon did." This ending does not work. A moral appeal individualises a structural problem, it asks one actor to bear a cost the structure distributes.

The problem is therefore not merely that people need more courage. It is that the environment needs to stop making courage individually punitive.The actor is caught between two incompatible demands. So, this is not about failure of character in decision makers. It is about a contradiction built into the environment. And this is what makes regulation necessary.

We tend to think of regulation as a brake: something that limits what companies are allowed to do. But, we need to take into account that regulation makes us capable of choosing. You cannot argue against a condition on the grounds that it constrains you, when the condition is what allows the choice you say you want. 

If every major actor knows that slowing down will be reciprocal, then slowing down no longer means handing the future to someone else.

Coordination changes the decision architecture.It makes the action that everyone says they want individually survivable. That is why pacing agreements, shared safety thresholds and credible regulation matter. They do not simply restrain the actors. They alter the payoff structure that makes restraint irrational.

We often think institutions fail because people ignore warnings. Sometimes they fail because the warning produces no viable action. And when that happens repeatedly, intelligence can make the problem worse because intelligent people are exceptionally good at explaining why continuing is reasonable.

Every person in the room can have a defensible reason. Every institution can have a rational explanation. And the system can still be moving in the wrong direction.

We have always known that knowing and doing are different capacities. What we have not always understood is that the distance between them is sometimes created not inside the individual, but around them.

A warning changes behaviour only when the environment makes a different behaviour possible.

We keep asking intelligent people to make better decisions. Perhaps we should also ask them to build environments in which better decisions can survive, because knowing fails to produce action when the environment makes action individually costly?