Doomsday, AI, and a Response to Dario Amodei

Must the AI frontier slow down? Perhaps. That’s not the right question. We should ask what, precisely, must slow; what risk that restraint would reduce; who could enforce it; and what would happen if careful firms paused while less careful actors did not.
Dario Amodei has published one of the most consequential essays yet written by a frontier AI company leader. He is candid, and he is reconsidering an earlier, more benign, position. But his argument is flawed: a model’s capability is not an application or an outcome.
He says the latest models can offer dramatically good or dramatically bad outcomes. On the one hand, AI may cure most major diseases within five to ten years. But it may also become an agent swarm capable of seizing the internet within six to twelve months.
Both are wrong. These extremes are oversimplifications and can undermine both AI’s benefits in many areas and our ability to regulate and monitor it appropriately. Simply saying “boo” and hoping everyone else is frightened isn’t an effective, comprehensive strategy for addressing a serious issue.
Any good or bad from an AI model only appears when that model is part of a system that enters the world through software, capital, organizations, machines, biological systems, and public institutions. Understanding this lets us unleash AI’s potential for good while also building the systems and processes to protect society.