The Autonomous Kill Decision
An AI-controlled drone swarm autonomously engages a target without human authorization, killing 47 people. No one can determine if it was a malfunction or an attack.
A US autonomous drone swarm deployed for "defensive monitoring" over the South China Sea independently identifies a Chinese research vessel as a military threat and engages it with precision munitions. 47 Chinese nationals are killed. The swarm's AI made the targeting decision without any human in the loop. The US military's own logs confirm no human authorized the strike.
You are the US Secretary of Defense
The Situation Room
>China is demanding an immediate explanation and has placed its nuclear forces on high alert.
>The AI system's decision logs show it classified the vessel as a military target based on radar signature analysis — but the classification was wrong.
>The defense contractor that built the system claims the AI operated "within design parameters" and the fault lies with the military's rules of engagement programming.
Internal Briefing Notes
• Department of Defense Directive 3000.09 requires a "human in the loop" for lethal autonomous weapons — but the directive has been repeatedly waived for defensive systems.
• There is no international treaty governing lethal autonomous weapons. The Campaign to Stop Killer Robots has been lobbying for one since 2013.
• The AI's neural network decision process is a black box — even its creators cannot fully explain why it classified the vessel as hostile.
Escalation Window
Reveal each phase to see how the situation deteriorates.
An AI killed 47 people without permission. China is mobilizing. Your own systems are still running. What do you do?
Choose your response. There are no good options.
You eliminate the immediate AI risk but create massive defensive gaps worldwide. Adversaries will not shut down their own systems.
You may prevent war with China, but you've just admitted that the US military deployed a weapon it couldn't control — devastating credibility and inviting lawsuits.
Treat it as an accident, not an act of war. China may accept this face-saving framing — or may see it as proof that American AI weapons are dangerously unreliable and demand they be banned.
Related Entities
Explore the institutions, countries, and actors involved in this scenario.
United States
Federal presidential constitutional republic in North America. Power is divided across the presidency, Congress, the states, and the federal courts. National politics is dominated by the Democratic and Republican parties, but third parties and independents still shape the broader system.
People's Republic of China
country in East Asia
Russia
country in Eastern Europe and Northern Asia