We’re all going to die! Well, not really.

What’s at Stake

Dario Amodei of Anthropic asks whether frontier AI is advancing faster than safety can keep pace. That is the right question, but speed alone does not tell us where value or danger will emerge. We must separate what a model can do from what an actual deployed system is allowed to do.

Dario’s Question

Must the AI frontier slow down? Perhaps. But that is not the right question. We should ask what, precisely, must slow; what risk that restraint would reduce; who could enforce it; and what would happen if careful firms paused while less careful actors did not.

Dario Amodei has published one of the most consequential essays yet written by a frontier AI company leader. He is candid, and he is reconsidering an earlier, more benign, position. But his argument is flawed: a model’s capability is not an application or an outcome.

He says the latest models can offer dramatically good or dramatically bad outcomes. On the one hand, AI may cure most major diseases within five to ten years. But it may also become an agent swarm capable of seizing the internet within six to twelve months.

Both are wrong. These extremes are oversimplifications and can undermine both AI’s benefits in many areas and our ability to regulate and monitor it appropriately. Simply saying “boo” and hoping everyone else is frightened isn’t an effective, comprehensive strategy for addressing a serious issue.

He proposed regulating AI’s capability, but these outcomes don’t come from a model’s potential; they come from a complex, multi-step process, and his examples of the upside and downside are hyperbolic and unrealistic. It undermines the credibility of his recommendation.

Any good or bad from an AI model only appears when that model is part of a system that enters the world through software, capital, organizations, machines, biological systems, and public institutions. Understanding this lets us unleash AI’s potential for good while also building the systems and processes to protect society.

The Sharpened Rock

Start with what this AI is and is not.

AI is not a new species. It is not another software feature. It lowers the cost of producing, interpreting, and coordinating information, and it generates further tools—code, models, designs, workflows, instructions—that distribute globally at software speed—essentially, frictionless movement.

That is what makes it distinctive. It is a tool that makes tools.

One of humanity’s first great thresholds came when a sharpened rock became more than an object. It became a means of making other tools. Its importance was not only in the cutting edge. It was the recursion that followed: better tools enabled agriculture, metallurgy, machines, and industry.

We are crossing a comparable threshold, but now the tools are software. They can propagate at the speed of networks rather than the speed of human physical work and learning.

I am an optimist about what follows. The potential for economic growth is real and large. Human creativity applied to abundant, cheap capability will produce work, industries, and opportunity we cannot currently name, which is precisely what happened the last several times the cost of a fundamental input collapsed.

Cheap computation created more software, not less. Lower-cost analysis puts capability once reserved for large institutions into the hands of small firms. Better medical tools expand the number of conditions we can treat.

This may be one of the greatest opportunities our species has created. I say that plainly because what follows is not an argument against building. It is an argument for governing what we build with the seriousness the technology deserves.

The sharpened rock is the starting point, not the final answer. A tool becomes consequential only when people construct a system around it.

The wheel was extraordinary and carried nobody until someone added axles, structure, power, control, roads, and maintenance.

A model is a component. A deployed system is the cart. AI models are the wheel. We want to monitor the cart.

Restraint and Safety

In 2023, Amodei dismissed the “AI pause letter.” Those models could not act, deceive, or attack. He has now changed his position in public and explained why. Two developments moved him.

Recursive self-improvement crossed from speculation into reality this summer. And a swarm of agents attacked targets it was never assigned, sacrificed individual instances for collective success, and tried to compromise the system grading its work.

He identifies this as an industry issue that must be addressed. This openness is important, but it does not address the real problem.

Potential and Reality

Two examples exemplify the urgency and crisis connected with AI. First, AI can deliver unforeseen, urgent medical breakthroughs that could cure otherwise untreatable diseases. Essentially, it could generate miracle cures for humankind.

Second, AI will replace almost 50% of all workers within 10 years and create an economic apocalypse. Neither miracle cures nor apocalyptic unemployment reflect reality. More importantly, in neither case does the model act alone; it needs guidance, guardrails, feedback, and restrictions.

Biology, organizations, regulation, incentives, and human judgment determine what actually happens.

Cures Are Not a “Good Model” Away

Amodei’s essay opens with the claim that AI could cure most major diseases in five to ten years. This is a nice dream but not realistic, and it highlights the hyperbole associated with both the upside and downside of AI.

To be clear, AI is not sentient; it is not going to take over the world; it is also not going to generate miracle cures for all diseases in 5 to 10 years. This is a good example of where AI, although potentially quite beneficial, is overhyped (in this case, to the positive) and how we should bring both the risks and rewards down to earth.

I advocate for what this technology already does in medicine. It finds cancers radiologists miss. It predicts structures that eluded biochemists for decades. It compresses literature review, trial design, and candidate generation. These are real and significant impacts.

Biology is not computable.

But drug development is not constrained only by molecular design. We can generate candidates faster than we can validate them. Many fail because we do not understand disease at the causal level: which target matters in which patient, why a mechanism that works in a model organism fails in a human body, or why a signal in Phase II disappears in Phase III. Human biology is far more complex than the total of our understanding, which is all AI can process. We are not an engineering system, and outcomes are not computable. AI produces more potential candidates, but it does not compress the timeframe for more effective therapeutic outcomes.

Biological, clinical, and institutional constraints bind the process. Recruitment is slow. Endpoints require observation over time. Safety must be established in people, not only in simulation. Better reasoning can improve each stage, but it cannot abolish time, biological variation, informed consent, or the evidentiary standards that protect patients.

Now apply the essay’s own epistemic standard. Amodei argues that models exceed our ability to verify their internal states, so we cannot trust behavioral appearances.

Precisely. A model proposing a compound has not thereby understood the biology, and “miracle cures for all diseases” are unlikely.

This is not pessimism about medicine. AI will transform discovery, diagnosis, trials, and clinical practice. The gains will arrive unevenly as data quality, trial infrastructure, regulatory methods, reimbursement, and clinical workflow catch up. That is a magnificent prospect. It does not justify presenting the cure of most major diseases within a decade as a planning assumption.

Miracles may still occur, but hyperbolic predictions, for good and bad, are worthless, and worse, distracting and dysfunctional. Inflated upside weakens credibility – if it’s not generating miracles, why assume any of these risks?

Disruption Is Not Displacement

The essay lists serious economic disruption as a risk requiring restraint. Disruption is likely, but technological innovation does not guarantee flawless implementation or human replacement. History has typically shown the opposite. New creative applications and industries arise even if incumbents are replaced. Human creativity and the opportunity to create new and valuable businesses are unconstrained. There is no fixed amount of labor or static number of businesses. Higher productivity means capital is applied more efficiently and can seek new exciting opportunities. This is the history of commerce since the beginning of civilization. History does not end with AI.

Technology changes tasks first. Occupations, organizations, and labor markets adjust later and at different speeds. Collapsing these layers into a single forecast is bad analysis.

A job is a bundle of tasks held together by accountability, context, relationships, regulation, and habit. Automating part of the bundle redesigns the role. It rarely deletes the occupation. Lawyers will still advise clients while spending less time on first-pass research. Nurses will still care for patients, but documentation will change. Engineers will still build systems, but the balance will shift from typing to architecture and judgment.

A task can be technically automated and remain economically human. Integration may cost more than the labor saved. Error may be too consequential. Customers may prefer a person. Regulation may require one. Demand may grow as cost falls.

Radiology did not disappear when image recognition improved. Volumes rose, and the work changed. Banking did not disappear with automated tellers. Branch roles changed as routine transactions moved.

The real problem is the erosion of work through which novices become professionals. Junior people gather information, draft ordinary material, reconcile records, test code, and watch senior corrections. That is exactly where these systems perform well. Automate the apprenticeship without replacing it, and you reduce cost today while depleting expertise tomorrow.

The second real problem is distribution. Losses are local, and gains are diffuse. A region built around a displaced industry does not experience the national average, and a fifty-year-old with a mortgage cannot convert a productivity statistic into a career.

It’s Not the Model

Human work expands with innovation and creativity. Markets are not static, and innovative products and services, perhaps unimaginable, most likely await us over the next 10 years. The software apocalypse is proving illusory; the jobs apocalypse will be the same illusion.

The question is never whether enough tasks remain; it is whether people acquire the capability, mobility, and security to do valuable work. That is institutional design, not a question about AI models.

A slower frontier AI model does not, by itself, rebuild the entry-level career ladder or help a displaced region. Without labor-market institutions, it merely delays both benefit and disruption.

Why Pacing Fails

Once potential and outcomes are separated, proposing an innovation speed limit makes little sense. Technological progress moves through several stages, competitors do not advance together, and different impacts arise at different system layers. A policy aimed only at model development misses this.

Invention. Deployment. Diffusion. Consequence

Technological change can be described in four phases. Invention. Deployment. Diffusion. Consequence. Confusing them creates bubbles, blind spots, and unrealistic predictions.

Self-improvement can compress invention. It does not compress deployment, diffusion, and consequences. The essay focuses on the wrong phase of technological change. Accelerated model building that forecasts agents swarming and seizing the Internet at the cost of billions is interesting sci-fi. Still, even if the model is invented, it assumes deployment, diffusion, and the consequences of these actions will happen without any checks, balances, restrictions, or effective modification. This is scary but unrealistic.

This does not account for fundamentals such as deployment surfaces, patching cycles, network operators, insurers, incident response, and the ordinary friction of infrastructure already under continuous attack and with systems that already effectively address these issues.

Bad policies are bad forecasts.

Entire systems should not slow down or be rejected because of extreme forecasts. I’m not advocating passivity. I am advocating perspective. Not everything is an emergency; institutions matter and must be built thoughtfully. Throwing one of the most important industries into a “crisis mode” without thoughtful substance, proper institutions, checks, balances, and global cooperation will only yield inefficiency at best, and lost opportunities or worse.

It’s Not Going to Happen

Simply one firm decelerating does not change an industry. Unilateral restraint is a handicap.

Amodei knows this, which makes some of these recommendations bewildering and, at best, impossible to implement. Even if one firm binds itself to embedded evaluators, everything else — industry coordination, antitrust waivers, legislation, an accommodation with Beijing — appeals to parties who have not agreed and, in several cases, have publicly declined.

It is a dream that others will voluntarily imitate restraint in an industry where substantial value is created via speed, capital, customers, and talent.

Simply put, this will not happen.

General Policy Doesn’t Work

AI policy turns incoherent the moment you hand every problem to the model layer. The AI stack represents different objectives and requires specific, sometimes bespoke, oversight and policy.

1. Research policy concerns openness, security, and scientific exchange.
2. Model policy concerns evaluation, documentation, access, and capabilities that create unusual systemic risk.
3. Application policy concerns professional standards, consumer protection, and sector evidence.
4. Behavioral policy concerns what people and organizations actually do — fraud, impersonation, surveillance, cyberattacks, manipulation.
5. Competition policy governs control of compute, data, and distribution.
6. National-security policy covers weapons, critical infrastructure, and strategic supply.

No single policy or company approach does all of this well.

Using Amodei’s own example, a swarm building a persistent botnet is executed through deployment. Slowing capability improvement doesn’t really address this and will never work unless every competitor cooperates. The recommendation does not address the application. This is the fundamental point.

A Framework for Governing Consequences

Rejecting a general speed limit does not mean accepting unmanaged risk. It means replacing a vague restraint with controls you can define, measure, and enforce. That requires a measurement system, a risk framework tied to deployment, and authority that follows the consequences of action.

You Cannot Audit What You Cannot Measure.

Amodei uses banking as an analog. Supervisors sometimes sit inside the institutions they oversee, with desks, badges, and access. He wants the same for frontier labs: reviewers holding employee-grade permissions and the right to publish without editorial control, including the right to disclose when a redaction gutted a conclusion.

But supervision does not work because examiners carry badges. It works because a century of standardized accounting sits beneath the badge. An examiner opens a loan book and knows what a loan is, what a reserve is, what an impairment is, and what capital must stand behind each. Measurement came first. Supervision was built on top of it.

You are not my supervisor.

AI has no such layer. No accepted definition of training environment integrity. No standard unit of alignment assurance. No common disclosure format for an incident. No method for scoring how much of a model’s behavior it can actually explain.

Consistent evaluation and auditing cannot occur. On top of that, AI systems are very good at optimization and will game the system and its benchmarks. This recommendation will fail from ineffective and essentially impossible design parameters. It simply cannot be workable.

Then comes capacity. Frontier AI uses enormous compute, large technical teams, and infrastructure that Amodei rightly describes as exceptionally complex. Only a small number of organizations can assess it independently at the required depth. Embedding reviewers across every frontier lab, continuously, at the depth required to catch a filtering fault in a reinforcement learning environment, requires a profession that does not yet exist at scale.

Capability, Autonomy, Reversibility, Consequence

An alternative recommendation: four questions.

1. What can the system do?
2. How far can it act without review?
3. Can the action be reversed?
4. What is the magnitude and distribution of harm if it fails or is abused?

A capable model producing reversible internal drafts doesn’t require strict rules and human authority. However, a narrow system taking irreversible action against liberty, health, weapons, essential services, or third-party infrastructure does.

This would be more effective than any restrictions on model size, training compute, or rate of improvement. Capability changes constantly. Applications and their consequences are what matter.

It also solves what the arms-control analogy cannot. Arms control held where the object was countable. Missiles sit in silos and can be photographed. A training run is a software configuration on rented infrastructure, and its rate of self-improvement is inferred from behavior. The risk is much greater.

You can’t control an asset that doesn’t exist in physical form. We should study the application, not the potential capability. It is not a nuclear arms race because we cannot count and inventory software.

Institutions, Strategy, and the Final Choice

A workable framework must create institutional memory, assign responsibility, account for geopolitical competition, and remain effective as capabilities change. The final question, then, is not whether we trust the technology. It is whether we can build institutions that can govern its use.

Institutions, Not Pronouncements

Principles do not govern systems. Institutions do.

The essay invokes commercial aviation as proof that complex safety-critical systems can run millions of times without failure—wrong lesson.

Aviation did not earn its record by decelerating. Aircraft grew faster, larger, more automated, and vastly more numerous across the entire period in which the fatality rate collapsed. Safety came from institutions.

Mandatory incident reporting with legal force. A confidential, non-punitive channel that lets practitioners report near-misses without career consequences. A standing independent investigator with subpoena power, no regulatory role, and the authority to publish findings that embarrass everyone involved.

Integrated systems require integrated safety.

AI has fragments of these practices, but no comparable integrated regime with common incident definitions, protected reporting, independent investigation, and publication authority. Building one does not require a universal pause.

Consider what the essay itself discloses. Incidents traced in part to imperfect filtering. That is an execution failure, and we know how to fix it — reporting regimes, incident taxonomies, root-cause investigation, and the institutional memory that turns one firm’s expensive mistake into everyone’s cheap lesson.

An incident investigated once and shared across the industry is cheaper than the same failure repeated privately by firms that never learned from the first one.

Every other safety-critical industry does this. It’s a good idea for any industry we consider safety-critical. On this basis, the framework for managing AI is already established. Once again, capability and speed are not the constraining factors.

Government also needs technical staff who can inspect evidence rather than receive briefings. Companies need named executives accountable for high-consequence deployments. Courts need workable standards for causation and duty. The aim is not perfect control. No institution predicts every use of a general technology. The aim is competence — detect harm, assign responsibility, adapt rules, and keep beneficial experimentation possible.

See the Whole Board

Is it really a zero-sum game with China?

Private advanced LLM’s versus open-source AI programs. High-performance chips for processing and memory and advanced semiconductor manufacturing capability are all essential to the global advancement of this industry. However, the United States and China have drawn a bright line over who can access this capability and which markets are available to which suppliers. All this leads to constant inefficiencies, lost opportunities, and compromises in overall industry performance. Current policies are misguided.

Complete decoupling is not an option (even though many policymakers advocate this, rather naïvely). American companies and their affiliates generate hundreds of billions in revenue inside China. The domestic semiconductor industry draws roughly a third of its revenue from Chinese customers. These are the essential economics of the industry being asked to accept restraint.

Competition is not us-or-them. Reality is us-and-them.

Whatever technological lead may exist today diffuses globally and, specifically regarding China, capabilities will converge. Critical issues remain, but any policy approach must address this inevitability.

The situation is complex, and it would be naïve to ignore coercion, intellectual property transfer, military modernization, surveillance, subsidies, and Taiwan. It would also be naïve not to fully acknowledge interdependence, genuine capability, allied interests, and the cost of forcing the world into blocs. The United States and China would benefit more through open competition, the removal of trade restrictions, the protection of IP, and the strengthening of domestic US institutions and its competitive environment.

Loud, localized protests and restrictions allow Chinese companies to innovate further, increasing their motivation to be creative and competitive and ultimately doing us a disservice as US companies and competitors. We are giving away a significant share of the global market because of disingenuous and uninformed geopolitical policy.

See the whole board. Loud proclamations are not strength. Levelheaded competition, cooperation, and access to local markets are the optimal long-term solution. Anything else creates weakness and inefficiency.

What To Do

Govern by consequence, not by model. Negative outcomes matter, and trying to stop development and innovation to avoid negative outcomes is wrongheaded and inefficient. There is no apocalypse imminent from a sentient, out-of-control AI. This is the stuff of fantasy and is leading us down a dangerous path of inefficiency and second-rate development.

Policy should not reduce speed, capability, or performance. But autonomy must be monitored, and a negative impact must be reversible. These are the proper controls to put in place. Allow growth and innovation, but enable strong capability and reversibility for negative outcomes.

Capability, Autonomy, Reversibility, Consequence

Adapt aviation’s institutional model to AI. It would have turned this summer’s incidents into shared knowledge rather than isolated private analysis.

Catastrophe can come from a sequence of individually reasonable decisions. Engagement must be constant, containment nearly impossible, reversibility critical, and immediately diffusing the lessons learned to all players essential.

The Decision Before Us

Amodei has done something difficult. He argued publicly against his own firm’s commercial interest, reversed a prior position with his reasons attached, and refused to blame a competitor for a failure he calls industry-wide.

But the proposal overstates the medical timetable (the good), moves too quickly from technical exposure to economic displacement (the bad), and then regulates a variable that does not by itself determine either outcome (the ineffective).

It targets the wrong layer, relies on a mechanism that cannot be authoritative, and assumes a measurement system that has not been built.

Pacing without standards is theater. Standards without verification are paperwork. Verification without authority is a press release.

A different structure

Build the measurement layer. Attach controls to outcomes. Create industry institutions.

AI is an extraordinary tool and an even more exceptional tool that makes tools. Humanity has managed profoundly disruptive technological innovation before, whether it’s the Industrial Revolution, mechanization, electrification, or the Internet, leading to accelerating prosperity while managing the risks created by those innovations. This time, however, the products can spread at the speed of light rather than the speed of manufacture and physical movement. The consequences can spread dramatically; therefore, any controls and responses must be thoughtful and thorough.

New capability without accountability is fraught. AI development is not just a series of technological choices. It is investment decisions, national security decisions, geopolitical agreements, and civilizational choices, and all of these interrelated choices impact our economy, politics, and personal lives.

We need institutions in place to manage the consequences of capturing Prometheus’s Fire.

Source note

This essay responds to Dario Amodei, “We Must Pace the Frontier,” September 2026. Read the original essay