AI systems can fail or cause harm in unexpected ways—from leaking sensitive data to generating biased outcomes to being controlled by hackers to “going rogue” of their own accord. Yet most organizations that develop and/or operate AI systems have no plan for dealing with an AI incident.

How would your business detect a cyberattack against one of your agentic AI systems? And how would you respond? Do you know what risks you face from using AI? This article shares key points to help business and technical leaders envision and implement AI incident response.

Key takeaways

• AI incident response goes beyond traditional incident response to account for AI’s unpredictable and complex failure modes and unfamiliar cybersecurity attack vectors.
• AI risk management, comprehensive AI system documentation, and the ability to monitor AI system behavior are prerequisites for AI incident response.
• The unique risks AI presents run the gamut from data loss to discrimination to privacy violations to psychological harm to copyright infringement.
• Getting started with AI incident response involves asking basic questions about who, what, when, where, and how.
• AI can play an important role in all forms of cyber incident response, but it remains prudent to keep humans in the loop to reduce the risk of AI failures.

What is an AI incident response plan?

An AI incident response plan is a formal process that guides an organization to more efficiently and effectively recognize, contain, manage, and remediate risks that manifest due to the performance or actions of an AI system. This includes cybersecurity attacks, ethical violations, legal issues, and other outcomes that cause negative or unexpected outcomes to individuals, organizations, and/or society at large.

AI incident response goes beyond traditional incident response to address AI’s unique and complex failure modes and cybersecurity vulnerabilities. Its goal is to minimize harm, accelerate remediation, maintain legal and regulatory compliance, and help reduce future AI risks.

Important elements in an AI incident response plan include:

• An up-to-date inventory of AI systems and their operational baselines
• The ability to monitor and detect unexpected AI outcomes or performance issues, including data leaks, biased or incorrect behavior, or cyberattacks.
• Procedures to isolate/“kill” affected AI systems, contain the spread of damage, and eliminate malicious intrusions.
• Steps to test and restore stable AI operation.
• Steps to capture root cause data to prevent recurrence.

What types of AI risk must we plan for?

AI systems require AI-specific incident response because they present unique risks and concerns that can be extremely difficult to predict, detect, or mitigate. Some examples include:

• AI systems can be extremely complex and opaque, making it problematic to identify where or why errors occur. Yet AI errors can proliferate exponentially, rapidly shifting from minor mistakes to major negative impacts.
• Many AI models are trained on historical data, which can become outdated over time. If not updated, AI outcomes can drift from actual conditions, resulting in invalid recommendations or outcomes.
• AI and machine learning systems are increasingly subject to emerging cyberattacks that differ from conventional attacks on traditional applications and networks. Many AI systems have been rapidly rolled out with scant attention paid to security vulnerabilities, giving hackers a wide-open attack surface.

Based on analysis of AI incidents reported in the AI Incident Database, the most common AI failure modes include cybersecurity attacks, unauthorized behavior, discriminatory results, lack of transparency in outcomes, privacy violations, threats to human physical or psychological safety.  Specific risks include:

• Operational risks from model hallucinations, model decay, rogue behavior, or cyberattacks (data poisoning, prompt injection, etc.)
• Compromising the civil rights of individuals, such as discrimination in AI-assisted hiring or lending.
• Privacy violations from improper handling of sensitive personal data.
• Product or professional liability claims resulting from property damage, physical and/or psychological harm, or contract/service level violations. Patient safety in healthcare settings can be a major risk.
• Supply chain risks arising from vulnerabilities in third-party components within the AI technology stack.
• Triggering compliance violations, including inability to meet incident notification requirements, insufficient model documentation, or failure to record incident details.
• Copyright or trademark infringement claims from generating inappropriate content.

How can we get started with AI incident response?

Start your AI incident response planning by answering recommended questions like these:

• How do you define an “incident” that will trigger the plan or not?
• When an incident occurs, what company policies and procedures go into effect?
• Who are the specific individuals and groups and associated roles and responsibilities with accountability for AI incident response?
• When an incident occurs, who should be notified, at what stage of the plan should they be notified, who should notify them, and how should they be contacted?
• Who is responsible to make decisions around shutting down systems or halting operations?

Like data breaches, ransomware attacks, and other traditional cyber incidents, AI incidents may involve your legal counsel, your cyber liability insurance provider, your IT service provider, and other familiar internal and external roles.  But AI incident response includes additional roles and has a wider overall scope because AI risks and failure modes are so diverse.

What steps should an AI IR plan address?

Key steps toward building an AI incident response plan include many that are part of traditional incident response:

• Put together an interdisciplinary team to manage AI risk. AI incidents are not just technology problems but can touch on many parts of the business and impact its financial, reputational, and legal standing.
• Develop a full inventory of AI systems in use within your organization, including identifying “shadow AI” services.
• Conduct risk assessment and threat modeling so you can account for foreseeable events.
• Prioritize the highest risk AI systems so you can focus limited monitoring resources.
• Create a general process framework for responding to unforeseen events.
• Document a baseline that describes correct operations for each AI system, so you can identify when something is wrong.
• Identify options for technical intervention to minimize harms or “flip the kill switch” on an out-of-bounds AI system.
• Educate and train staff on applicable policies and procedures to ensure plan awareness and proficiency.
• Identify all implicated vendors, partners, and other third-party stakeholders, and make sure your plan includes them.
• Establish procedures for monitoring AI systems, detecting incidents, and confirming that an incident has occurred.
• Establish procedures for collecting incident data, performing real-time notifications, and documenting response activities to improve performance.
• Conduct thorough testing and reviews of AI system failures, including third-party services and other dependencies, to identify and correct root causes.
• Take the opportunity to harden new AI system versions against cyberattack prior to relaunching.
• Document lessons learned.

How can AI help with cyber incident response?

Countering the velocity of emerging AI-enabled cyberattacks takes AI-assisted detection and response. Typical incident response often includes manual processes and human intervention to analyze data and identify threats, slowing detection times and limiting throughput.

AI can process large data volumes from multiple sources in real-time, rapidly ranking possible threats and potentially identifying patterns that humans could miss. AI can also learn and adapt as new attacks emerge, helping AI incident response to be more proactive. Another strength of AI in incident response is automating routine tasks like generating trouble tickets and alerts so humans can focus on more strategic activities.

Some of the benefits of using AI in cyber incident response include:

• Faster threat identification and anomaly detection.
• More efficient alert processing that is resistant to overload and alert fatigue.
• Better informed crisis decision-making due to automated, structured AI inputs.
  An automated threat response workflow to shut down incidents much quicker.
• An adaptive detection and response capability that is more robust against novel threats and attacks.
• More comprehensive threat intelligence data collection.
• Enhanced operational efficiency and scalability to help offset AI solution investments.

Despite the many advantages of AI-assisted incident response protocols, it is important to keep a human in the loop. AI behavior can be unpredictable and the consequences of AI agent failure in a cyber incident response scenario could be catastrophic.

What’s next?

For more guidance on this topic, listen to Episode 156 of The Virtual CISO Podcast with guest Jason Rebholz, CEO and co-founder at Evoke Security.