Chapter 47 — Failure and recovery trial
Lesson map
Lesson 47 published. Map, example, EX139.
flowchart TD
N0["Learning outcomes"]
N1["47.1 Trial design"]
N2["47.2 Running the trial"]
N3["47.3 Improving procedures"]
N4["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4
N4 --> EX["Worked example"]
EX --> Q["Exercise EX139"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["47.1 Trial design"]
N2["47.2 Running the trial"]
N3["47.3 Improving procedures"]
N4["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX139"]
[Learning outcomes]
|
v
[47.1 Trial design]
|
v
[47.2 Running the trial]
|
v
[47.3 Improving procedures]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX139]
Learning outcomes
You will (1) rehearse failure and recovery on the test network; (2) measure recovery; (3) improve procedures.
47.1 Trial design
Before relying on recovery procedures, Jonas runs a failure and recovery trial on a full copy of the system on the test network. Scenarios include the loss of two validators, a corrupted gateway database and a mistaken fund parameter change detected during its timelock. Each scenario has a start signal, expected actions, success criteria and a time target. Participants include the developer, two board keyholders and a partner operator. Trials expose gaps that tabletop exercises cannot. The scenarios are approved by the board.
47.2 Running the trial
Restoring two validators takes three hours, within the four-hour target, but reveals that one partner's restore guide is out of date. Restoring the gateway database from backup takes 50 minutes and a replay of queued readings succeeds. The mistaken fund parameter change is detected by the monitor within 40 minutes and cancelled within the timelock by the multi-signature wallet. Jonas records every action and time, and notes where people hesitated. He also notes that the trial took longer to start than planned because participants were not sure who should declare the scenario started.
47.3 Improving procedures
After the trial, the partner updates its restore guide, the gateway backup schedule is increased to every four hours, and the cancellation procedure gains a checklist because one keyholder was unsure which transaction to sign. Jonas reruns the cancellation scenario a week later and it completes in 15 minutes. Recording measured recovery times gives the board realistic expectations and evidence for regulators and members. He also adds a named trial lead role to future trials so that start and end are declared clearly, and schedules the next full trial for six months later.
Worked example
The trial restores two validators in three hours, the gateway database in 50 minutes, and cancels a mistaken change within the timelock; three procedure improvements follow.
Synthetic case. Scenarios, timings and participants are synthetic.
Errors and limits
Assuming backups and procedures work without trial leaves recovery untested until a real emergency.
Exercises
- EX139 — Validator restore time. How long did restoring two validators take? Rubric: 3 = three hours;; 2 = 3 h;; 1 = four days;; 0 = none.
- EX140 — Detection time. How quickly was the mistaken change detected? Rubric: 3 = within 40 minutes;; 2 = 40 minutes;; 1 = never;; 0 = none.
- EX141 — Rerun result. How long did the rerun cancellation take? Rubric: 3 = 15 minutes;; 2 = quarter hour;; 1 = hours;; 0 = none.
Synthesis
Measured recovery trials turn procedures into proven capabilities and guide improvements.
Continue studying — chapter 47 of 50
← Chapter 46 (draft) · Course chapter index · Chapter 48 (draft) →