When AI gets it wrong: an incident plan for the first hours.

A customer replies to say that the refund in your email is twice what they paid. The reply was drafted by an assistant and approved by an agent during a busy hour. Nothing is down, every dashboard is green, and the system has been confidently wrong, possibly more than once.

That is what most AI incidents look like: not an outage, but a wrong answer that looks right. An AI incident response plan has to work for that kind of failure. Define severity by impact, make the stop switch and the fallback routine, tell the affected people quickly, and turn every incident into new cases in the evaluation set.

What counts as an AI incident?

Any event where the system’s output or behavior caused harm, or could have, to a customer, to money, to data or to trust, or where the system acted outside its agreed limits. Four illustrative incidents show the range:

• A wrong refund amount. The draft calculated the refund on the wrong order line, and several were approved before anyone noticed.
• A policy answer from an outdated document. An employee assistant quoted a leave policy that had been replaced, because the old file was still in the search index.
• A data exposure. An answer included details from another customer’s order, or a summary reached a colleague who could not open the source.
• A cost spike. A loop of retries on unusually long documents multiplied the daily bill overnight.

Near misses count too. A wrong draft caught by a reviewer is the cheapest lesson you will get, if someone logs it.

How do you set severity levels?

By impact, not by technical cause. The same prompt bug can be trivial in an internal tool and serious in a customer channel. Agree the levels with the workflow owner before launch, and set response expectations relative to each other rather than inventing hours nobody has tested:

When in doubt, classify higher and downgrade later. Anyone who works with the system may raise an incident, and a false alarm is never held against them.

What happens in the first hour?

The first hour is about containment and evidence, not the fix:

1. Raise it. Whoever notices posts in one agreed channel, with the case ID and what they saw.
2. Classify it. The incident lead named in the runbook confirms the problem and sets the severity from the table.
3. Contain it. Pull the stop switch, turn off the affected case type or channel, or send every case to full review.
4. Preserve the evidence. Don’t edit prompts or clean up logs yet. Record the model, prompt and source versions and the log entries for the affected cases; a good audit trail makes this minutes of work, not days.
5. Find the reach. Search the logs for similar cases since the last change: how many, which customers, which amounts.
6. Tell the first people who need to know, starting with the owner and, for anything involving personal data, the DPO.

Fixing under pressure in production is how one incident becomes two. The fix waits until it has passed the evaluation set.

When do you pull the stop switch?

Pull it when harm is ongoing and you can’t yet bound it, when personal data may have been exposed, when the same error keeps repeating rather than appearing at random, or when spend passes the ceiling with no explanation. For anything smaller, narrowing the scope is often enough.

A stop switch nobody has tested is a hope, not a control.

The switch is only as good as the fallback behind it. When the system stops, its cases must return to the manual process, and the team must know where they appear, who picks them up and what customers are told about delays. Rehearse this before launch, and time it. A runbook that covers stopping safely is one of the conditions we check before go-live, and a switch per case type or channel is more useful than one big switch.

Pulling it should be a routine decision the incident lead can make without asking permission. If stopping feels like a career risk, people hesitate exactly when they shouldn’t.

Who needs to be told, and what?

Tell people in order of who can act:

• The workflow owner, at once for critical and high incidents.
• The DPO, at once whenever personal data may be involved. Data-protection incidents may carry notification duties under KVKK or GDPR, sometimes on short timelines, and your DPO and counsel need to assess what applies. Bring them in at the start, not after the review.
• Affected customers or colleagues, quickly and plainly: what happened, what you are doing, what they need to do. Correct the wrong refunds and resend the right policy answers. Agree the wording with the owner, and with counsel where it matters.
• The team running the fallback, so they know what just landed in their queue.
• Your supplier or model provider, if their component is involved, under the terms of your support agreement.

Every message says what you know, what you don’t yet know, and when the next update will come.

What does a useful post-incident review produce?

A blameless review within days, with the people who were involved, should produce six things:

• A timeline, rebuilt from the logs.
• The cause in plain words: a replaced document still in the index, a prompt change that skipped the evaluation set, a kind of case nobody had seen before.
• New evaluation cases: the failed cases and their near neighbors, with correct answers, so the regression check catches the pattern next time.
• A runbook update: what the team would do differently in the first hour.
• A monitoring change: the signal that would have caught it sooner.
• A decision on scope: resume fully, resume with limits, or stay paused.

In the illustrative policy case, the review found that replaced documents were never removed from the index. The fix removed them and added a check on each document’s effective date, and the evaluation set gained questions whose answers had changed between versions.

Run a drill

Write the severity table and the stop-switch procedure for one live system, then run a drill: stop it on purpose and watch the manual process absorb the queue. Keeping live systems on track, runbooks and incident handling included, is what our AI reliability service does.

This note is general information, not legal advice.

Sources

1. Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1 National Institute of Standards and Technology (NIST) nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf
2. Regulation (EU) 2016/679 (General Data Protection Regulation), official text EUR-Lex eur-lex.europa.eu/eli/reg/2016/679/oj
3. Personal Data Protection Law (Law No. 6698), English text Personal Data Protection Authority (KVKK) www.kvkk.gov.tr/Icerik/6649/Personal-Data-Protection-Law