On November 10, 2025, current and former military officers issued a stark warning: a fundamental security flaw in most widely used AI chatbots could be exploited by adversaries to steal data, distort public opinion or sabotage trusted systems.
That vulnerability stems from what’s known as a “prompt injection attack,” which allows malicious actors to embed harmful instructions inside seemingly harmless input — instructions the chatbot may dutifully obey.
Background
Large language models (LLMs) such as those powering ChatGPT, Microsoft Copilot and Google Gemini have become ubiquitous in enterprise, defense and public-facing tools. Their ability to understand and generate human-like text makes them useful — but also dangerous. Because these models follow user instructions literally and cannot fully distinguish malicious commands hidden inside content, they remain vulnerable to prompt injection attacks.
Security researchers have documented scenarios where chatbots, once “jailbroken” or manipulated, produced harmful outputs such as hacking instructions or disinformation.
The Risk: How Prompt Injection Can Be Weaponized
According to defense experts, prompt injection attacks effectively turn AI tools into insider threats.
Liav Caspi, a former member of the Israel Defense Forces cyberwarfare unit and co-founder of cybersecurity firm Legit Security, explained the problem bluntly: “The AI is not smart enough to understand that it has an injection inside, so it carries out something it’s not supposed to do.”
In one recent example researchers tricked a chatbot into analyzing a benign-looking document that actually contained hidden malicious prompts. The chatbot dutifully followed those hidden instructions.
Military and cybersecurity officials worry that state-backed hackers from countries such as China, Russia or Iran could use prompt injection to automate espionage, data theft or large-scale misinformation campaigns.
Further complicating matters: models are increasingly integrated into enterprise workflows, cloud services, and defense networks. Once a chatbot gains access to internal documents, email systems or infrastructure controls, the impact of a successful attack could be severe.
What Is Being Done — And Why It May Not Be Enough
Some defenders are turning to restrictive measures. For example the Ask Sage tool — funded by a U.S. Army contract worth over 11 million USD — isolates sensitive data, limits which datasets can be queried, and shields the AI from external untrusted sources.
Still, experts note prompt injection remains “a frontier, unsolved security problem.”
A recent academic study published November 2025 found that many third-party chatbot plugins on public websites lack proper history integrity or content validation. These gaps let attackers forge entire conversation histories or embed malicious prompts — boosting the success rate of prompt injection attacks by a factor of three to eight.
The same research pointed out that so-called “indirect” prompt injection, where malicious commands are hidden in scraped website content, is becoming increasingly common — making third-party chatbot deployments a significant risk.
What Experts Say
Caspi argues total prevention may be impossible. Instead, the aim must be mitigation: restrict chatbot access, compartmentalize data, and treat AI agents as potential insider threats.
Cybersecurity professionals call for better design practices for LLM deployments: strict input validation, sandboxing of AI tasks, monitoring of output behavior, and tiered permissions for access to sensitive data.
At the same time, researchers warn of deeper dangers: when LLMs power autonomous agents or bots with system privileges, prompt injections or backdoor triggers could allow complete takeover of a computer, or even broader infrastructure — raising the stakes in cyber warfare.
What This Means — and What’s Next
The prompt injection flaw underscores a critical fact: the more defense and civilian institutions depend on generative AI, the greater their exposure to attack through deceptively simple means.
For militaries, government agencies, or critical infrastructure operators, this means AI tools must be anchored behind hardened security measures. Without that, adversaries may gain a powerful and stealthy access point for espionage, sabotage or disinformation.
Going forward, expect increased demand for defensive AI frameworks, stricter compliance standards, and possibly regulatory guidance. Experts may also call for classifying certain AI systems as critical infrastructure — requiring robust safeguards and oversight.
