Chapter 47 — Failure and recovery trial
Lesson map
Lesson 47 published. Map, example, EX093.
flowchart TD
N0["Learning outcomes"]
N1["47.1 Trial scenarios"]
N2["47.2 Objectives"]
N3["47.3 Running the trial"]
N4["Engineering note"]
N5["47.6 Following up the trial findings"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX093"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["47.1 Trial scenarios"]
N2["47.2 Objectives"]
N3["47.3 Running the trial"]
N4["Engineering note"]
N5["47.6 Following up the trial findings"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX093"]
[Learning outcomes]
|
v
[47.1 Trial scenarios]
|
v
[47.2 Objectives]
|
v
[47.3 Running the trial]
|
v
[Engineering note]
|
v
[47.6 Following up the trial findings]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX093]
Learning outcomes
You will (1) design and run a controlled failure trial; (2) recover from cryptographic failures within objectives; (3) measure recovery time; and (4) improve plans from results. It tests the resilience designed in Chapter 33.
47.1 Trial scenarios
In an isolated replica environment: (a) PQC library vulnerability requiring emergency patch and re-key; (b) LMS signing HSM restored from backup — confirm refusal to sign until state is reconciled; (c) simulated algorithm break: disable ML-DSA in policy and switch bills-of-lading signing to the alternative; (d) middlebox update that drops large ClientHellos at one site.
The four scenarios cover different capabilities: patching and re-keying test operations; the HSM restore tests state controls; the algorithm break tests agility; the middlebox change tests path-specific response.
47.2 Objectives
Define recovery time objectives: e.g. patch and re-key edge within 72 hours; switch signature algorithm within 14 days; path-specific exception for middlebox within 2 hours. Measure actual times.
The objectives come from requirements and risk: 72 hours for an edge re-key reflects the exposure of a public service, while 14 days for a signature switch reflects the need for partner verifiers to update. Setting them before the trial makes the results meaningful; objectives chosen afterwards tend to match whatever happened and teach nothing about readiness.
47.3 Running the trial
Brief participants; inject faults; observe without helping; record times, decisions and problems; stop if safety or production is at risk (it should not be — replica only).
Observers do not help because the point is to see what the runbooks and team can do alone. Recording decisions as well as times shows where people hesitated or improvised, which often reveals a missing runbook step. The stop rule is stated in advance even though the environment is a replica, so participants know it exists and practise using it.
47.4 Worked example
Model trial results: (a) 41 hours; (b) HSM refused, but the runbook lacked reconciliation steps — added; (c) 19 days, exceeding objective because partner verifiers needed updates — partner test endpoints added to the plan; (d) 35 minutes.
Three of the four scenarios met their objectives: 41 hours against 72, 35 minutes against 2 hours, and the HSM correctly refusing to sign. The signature switch took 19 days against 14 because partner verifiers had to update, a dependency that agility within BPA cannot remove. Adding partner test endpoints shortens that path for the next exercise.
47.5 Common errors and limits
- Trials on production.
- Objectives set after the trial.
- No follow-up actions.
Trials on production risk causing the very incidents they are meant to rehearse. Objectives set afterwards cannot show whether performance was good enough. Without follow-up actions, the trial produces a report and no improvement; at BPA each gap, such as the missing reconciliation steps, became a tracked action that was retested. A trial counts as complete only when every action has an owner, a date and a planned retest.
Engineering note
Trials should vary who is on duty. A recovery that works only when the system's designer is present is not a recovery capability; run at least one scenario with the on-call team alone and the designers observing silently. Measure not only elapsed time but also decision quality: whether the team consulted the CBOM, followed the runbook and communicated with partners at the right moments. Those observations drive the most valuable improvements.
47.6 Following up the trial findings
Each gap found in the trial becomes a tracked action: the missing reconciliation steps in the runbook, the partner test endpoints and the slower-than-planned 19-day partner rollout. The trial is repeated for the scenarios that failed their objective, and the repeat results are added to the evidence dossier beside the originals, so readers can see both the failure and the correction.
Keeping both results also shows assessors that the team corrected what failed. The repeat uses the same scenario and objective as the original, so the improvement is measured on equal terms. If the repeat still misses the objective, the gap is reported in the residual-risk statement rather than hidden.
Exercises
- EX093 — Scenario design. Design scenario (c) in detail with injects and observers. Rubric: 3 = realistic, measurable, isolated.
- EX094 — After-action. Write the after-action report for scenario (c). Rubric: 3 = root cause, actions, owners, dates.
Synthesis
Failure trials measure real recovery capability against objectives and turn surprises into plan improvements.
Continue studying — chapter 47 of 50
← Chapter 46 (draft) · Course chapter index · Chapter 48 (draft) →