AI Hallucination Nearly Triggers US Military Operation
techcrunch ·
Military aircraft were already airborne this spring when U.S. officials discovered that AI-generated hallucinated intelligence had helped drive an armed operation against a Chinese vessel; the operation was aborted at the last moment, averting a potential conflict with China, according to CNN. The intelligence report, circulated during the war with Iran, falsely claimed that the vessel was carrying components for a nuclear weapons program. A Special Operations Command analyst had asked an AI chatbot to combine open-source data with classified signals intelligence, but the chatbot misidentified the ship’s cargo manifest. The analyst then used the chatbot again to turn the erroneous findings into an official-looking summary that circulated through command channels. The incident highlights the risk that AI errors can move up military chains of command as the U.S. seeks to use AI to accelerate decision-making and its operational kill chain. GovAI scholar and former Army officer Jake Steckler said the episode should prompt stronger safeguards, warning that prioritizing rapid adoption could undermine service members’ trust in these systems and ultimately slow their use.
techcrunch