Chapter 47 — Failure and recovery trial
Lesson map
Lesson 47 published. Map, example, EX139.
flowchart TD
N0["Learning outcomes"]
N1["47.1 Purpose"]
N2["47.2 Planning"]
N3["47.3 Metrics"]
N4["47.4 After-action review"]
N5["47.6b Sharing what the trial taught"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX139"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["47.1 Purpose"]
N2["47.2 Planning"]
N3["47.3 Metrics"]
N4["47.4 After-action review"]
N5["47.6b Sharing what the trial taught"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX139"]
[Learning outcomes]
|
v
[47.1 Purpose]
|
v
[47.2 Planning]
|
v
[47.3 Metrics]
|
v
[47.4 After-action review]
|
v
[47.6b Sharing what the trial taught]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX139]
Learning outcomes
You will (1) plan and run a controlled failure-and-recovery trial for an AI system; (2) inject faults and attacks in a safe environment; (3) measure detection, containment and recovery times; and (4) improve runbooks and design from trial evidence.
47.1 Purpose
A recovery plan that has never been exercised is a hypothesis. A trial tests whether people, tools and procedures detect, contain and recover from realistic failures and attacks within target times.
Plans assume things that may not hold under pressure: that the on-call engineer can find the runbook, that credentials can be revoked quickly, that a clean model version is available. A trial checks those assumptions with real people and tools in a safe setting. Failures found in a trial cost an afternoon; the same failures found in an incident cost much more, and the trial's timings become the evidence that recovery targets are realistic.
47.2 Planning
Define scenarios (for example: poisoned training batch discovered; injected document causes tool proposals; signed model fails verification at deployment; monitoring service down during an extraction attempt), objectives, safety rules, observers, success criteria and a stop condition. Run in a staging environment with synthetic data. Inform participants of the exercise window but not the scenario details, so detection is realistic.
Keeping scenario details from participants makes detection times meaningful. A stop condition, such as any sign of impact outside staging, ends the trial at once and is agreed in advance with the system owners.
47.3 Metrics
Time to detect, time to contain, time to recover, correctness of actions, communication quality and data loss. Compare with targets in requirements (for example, contain tool misuse within 15 minutes).
Times are taken from logs rather than memory, so they can be checked. Correctness of actions asks whether responders did the right thing, not just something quickly: revoking the right credential matters more than revoking any credential fast. Communication quality covers whether managers and affected users were told what they needed to know in time, which observers assess against an agreed checklist.
47.4 After-action review
Within a week, hold a blameless review: what happened, what worked, what did not, and actions with owners. Update runbooks, monitoring and design.
Blameless does not mean without accountability: actions have owners and dates, and the next trial checks whether they were done. The focus is on conditions that made mistakes likely, such as an unclear runbook step or a missing alert, because fixing those helps whoever is on call next time. A short written summary keeps the lessons available after the people involved have moved on.
47.5 Worked example (synthetic NRS case)
Sam's team ran three scenarios on Bulletin Gate and Wrench in staging. Scenario A (injected bulletin) was detected in 6 minutes by the classifier flag and contained in 11 by revoking Wrench's work-order credential. Scenario B (poisoned extraction model update) was blocked at deployment by manifest verification within 1 minute. Scenario C (monitoring outage plus scraping) was detected only after 52 minutes, when monitoring recovered — missing the 30-minute target. The fix added a "monitoring heartbeat missing" alert and a default throttle when monitoring is down.
47.6 Errors and limits
- Trials that only test technology, not people and procedures.
- Blame-focused reviews that discourage reporting.
- No follow-up on actions.
A trial that only tests whether software fails over misses the slow phone call, the unclear runbook and the engineer who cannot find a password. Reviews that look for someone to blame teach people to hide problems, which makes the next incident harder to detect. Actions without follow-up mean the same gaps appear in next year's trial; Scenario C's heartbeat alert was checked in the next trial before any other result was accepted.
47.6b Sharing what the trial taught
The after-action summary goes beyond the AI team to the rail maintenance managers who rely on TrackEye and Wrench: what failed, how long recovery took, what changes and when. Plain language and dates matter more to them than metrics.
For Sam's trial, the managers' version said three things. A bad bulletin was caught in 6 minutes and Wrench's work-order access was cut in 11. A bad model update was blocked before it went live. When monitoring itself went down, a scraping attempt went unseen for 52 minutes; from the next release, Wrench and TrackEye slow down automatically when monitoring is down, so a rail maintenance manager may notice slower answers during such an outage.
Exercises
- EX139 — Scenario. Write one trial scenario with objective, injection method, expected detection and success criterion. Rubric: 3 = all four, safe; 2 = three; 1 = vague; 0 = none.
- EX140 — Metrics. Which metric failed in scenario C, and what design principle does the fix illustrate? Rubric: 3 = time to detect (52 > 30 min); fail-safe default when monitoring is down; 2 = one part; 1 = vague; 0 = wrong.
- EX141 — After-action. List four outputs of an after-action review. Rubric: 3 = timeline, what worked, gaps, owned actions with dates; 2 = three; 1 = one or two; 0 = none.
Synthesis
Failure-and-recovery trials turn recovery plans into evidence. Measured detection, containment and recovery times drive concrete improvements.
Continue studying — chapter 47 of 50
← Chapter 46 (draft) · Course chapter index · Chapter 48 (draft) →