False AI report nearly provoked an incident between the US and China
US military this spring began preparing an operation to intercept a Chinese vessel in the Middle East after receiving an intelligence report claiming it was carrying components for a nuclear weapons program. Shortly before the operation began, it turned out that a key conclusion of the report had been erroneously generated by artificial intelligence.
CNN reports this, citing four sources familiar with the incident. The incident occurred during the war with Iran and, according to one source, could have led to a dangerous escalation between the US and China.
After the intelligence message was disseminated, the US military began preparing to intercept the vessel. According to two CNN sources, armed service members were preparing to board it. Military aircraft were already in the air.
The operation was halted only after additional verification of the initial intelligence. It turned out that an analyst in a US special operations forces unit had used an artificial intelligence chatbot that incorrectly identified the cargo carried by the Chinese vessel. CNN could not determine what exactly was on board.
One source described the prepared document as "completely false." CNN emphasizes that a US operation against a Chinese vessel could have created a risk of direct armed conflict between the two countries.
AI combined classified and open data and made an erroneous conclusion
The initial information about the ship's manifest came from the US Special Operations Command Pacific, based in Hawaii. The analyst fed this information to a chatbot for analysis.
It is unknown whether he used a commercial AI system or one of the tools created for the US government.
According to CNN, the system combined open-source data with classified signals intelligence held by the US government and then reached an erroneous conclusion about the nature of the cargo.
The analyst then used the AI again to format the results into a standard intelligence document. It was in this form that the information was distributed among military structures and treated by recipients as a regular intelligence report.
No unified rules for verifying AI results exist across the US military
CNN sources said that various US military and intelligence community branches are rapidly adopting artificial intelligence, but in a decentralized manner. Different structures use different systems and their own safety guidelines.
As a result, the US military and intelligence community currently lack a uniform standard for verifying information obtained with generative AI.
Sources claim that similar AI "hallucinations" have already occurred in intelligence structures, and the case of the Chinese vessel was not isolated.
Particular concern is raised by the increasingly active use of AI in target selection and analysis. One CNN source said that the use of such systems in targeting is rapidly expanding, while clear rules about human oversight and prevention of erroneous strikes are still insufficient.
Pentagon accelerates AI adoption
The US military uses artificial intelligence for processing large volumes of intelligence, operational planning, logistics, resource management, and decision support on the battlefield.
In January, the Pentagon presented a new strategy for the accelerated adoption of artificial intelligence. Its directions include using AI for battle management, decision support, and accelerating the chain from target detection to engagement.
The document also envisions expanding access for military personnel and civilian employees of the defense department to advanced AI models. The Pentagon explains the acceleration by saying that the US cannot afford to fall behind potential adversaries, primarily China, in the use of AI in the military sphere.
However, the episode with the Chinese vessel demonstrates another risk: a dangerous decision may be made not because the AI acts autonomously, but because a person takes its erroneous output as reliable intelligence.
US Special Operations Command Pacific and the Pentagon did not respond to CNN's requests for comment on the incident.
Based on materials from: CNN, US Department of Defense
