AI hallucinations imperil critical infrastructure security

AA-Omniscience’s 2025 benchmark tested 40 language models and found 36 were more likely to produce confident but incorrect answers than correct ones on difficult questions. The report highlighted risks when those outputs are used in cybersecurity operations that support power grids, transportation systems and other critical infrastructure.

Hallucinations occur when a model generates plausible-sounding information that is not factually true. Models produce the most likely sequence of words from their training data rather than verifying facts, which can lead to fabricated sources, nonexistent studies or made-up data presented with an authoritative tone. When those outputs feed human decisions or automated systems, they can introduce operational risk.

The analysis identified three practical effects on cybersecurity. First, models can miss novel or underrepresented attack techniques that are not reflected in their training data, leaving zero-day attacks or unusual tactics unflagged. Second, models can produce false positives by misclassifying benign activity as malicious, triggering unnecessary incident responses, system interruptions and increased alert fatigue. Third, models can recommend incorrect remediation steps, such as deleting files or changing configurations; if those recommendations are executed, they can broaden an incident, cause data loss or enable lateral movement.

Factors that increase hallucination risk include flawed or outdated training data, overrepresentation of particular scenarios, the absence of built-in factual verification in base language models, and vague prompts that leave room for assumptions. The report also warned of a feedback loop in which AI-generated content enters public sources and then becomes part of future training datasets, potentially degrading model reliability over time.

A contributor at Keeper Security, Ashley D’Andrea, recommended operational controls to reduce harms. The suggestions include requiring human review before AI outputs trigger privileged actions, running AI systems with least-privilege access, auditing and updating the data used to train or ground models, training staff on prompt techniques to reduce ambiguity, and monitoring privileged activity for both human and machine identities.

The report noted that hallucinations cannot be fully eliminated but that technical controls combined with human oversight can reduce their impact as AI tools are integrated into infrastructure operations.