Chapter 47 — Failure and recovery trial
Lesson map
Lesson 47 published. Map, example, EX139.
flowchart TD
N0["Learning outcomes"]
N1["47.1 Planning the trial"]
N2["47.2 Running the recovery"]
N3["47.3 Learning from the trial"]
N4["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4
N4 --> EX["Worked example"]
EX --> Q["Exercise EX139"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["47.1 Planning the trial"]
N2["47.2 Running the recovery"]
N3["47.3 Learning from the trial"]
N4["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX139"]
[Learning outcomes]
|
v
[47.1 Planning the trial]
|
v
[47.2 Running the recovery]
|
v
[47.3 Learning from the trial]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX139]
Learning outcomes
You will (1) plan a large-scale recovery exercise; (2) recover the firmware root from custodian shares; (3) learn from exercise findings.
47.1 Planning the trial
Ridgeway holds an annual failure and recovery trial, and Aiyana helps plan this year's scenario: the primary signing HSM is destroyed in a simulated fire, and a security patch must be signed and released within forty-eight hours. The trial uses the real backup procedures but a test root and test meters, so no production key is exposed. Objectives, success criteria, observers and a stop rule are agreed in advance. Planning ensures the exercise tests procedures rather than improvisation. A safety officer may halt the trial at any time if a step risks real systems.
47.2 Running the recovery
Three of five custodians are called; one is abroad and one is unreachable, but the three-of-five quorum from Chapter 32 allows recovery. They meet in the backup site, restore the test root into a spare HSM from encrypted shares, issue a new intermediate in a ceremony and sign the patch. Test meters accept it. Total time is thirty-one hours. Observers record each step. One delay arises because the spare HSM's firmware was older than required, needing an update during the trial.
47.3 Learning from the trial
The post-trial review lists findings without blame: the spare HSM was not kept on the same firmware as production; the custodian contact list had an outdated phone number; and the ceremony script lacked a step for recording the new intermediate in the inventory. Each finding gets an owner and deadline. Aiyana updates the ceremony script and adds spare HSM firmware checks to the monthly operations checklist. The trial's value lies in these findings, discovered in practice rather than during a real emergency.
Worked example
The trial recovers a test root from three custodian shares, signs a patch in thirty-one hours, and yields three blameless findings with owners.
Synthetic case. The scenario and timings are invented.
Errors and limits
Exercises with unrealistic conditions give false confidence. Using production keys in trials risks real exposure.
Exercises
- EX139 — Test root. Why does the trial use a test root and test meters? Rubric: 3 = to exercise real procedures without exposing production keys;; 2 = safety;; 1 = realism;; 0 = none.
- EX140 — Quorum in practice. How did the quorum design help during the trial? Rubric: 3 = only three of five custodians were available, enough for recovery;; 2 = quorum;; 1 = not;; 0 = none.
- EX141 — Trial findings. Name two findings from the post-trial review. Rubric: 3 = two of outdated spare HSM firmware, wrong phone number, missing inventory step;; 2 = one;; 1 = vague;; 0 = none.
Synthesis
Realistic recovery trials with blameless review reveal the small operational gaps that would cripple a real emergency.
Continue studying — chapter 47 of 50
← Chapter 46 (draft) · Course chapter index · Chapter 48 (draft) →