Chapter 47 — Failure and recovery trial
Lesson map
Lesson 47 published. Map, example, EX139.
flowchart TD
N0["Learning outcomes"]
N1["47.1 Trial"]
N2["47.2 Measurement"]
N3["47.3 Issues"]
N4["47.4 Lessons"]
N5["Planning the trial safely"]
N6["Recording the timeline"]
N7["Extending the trial programme"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX139"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["47.1 Trial"]
N2["47.2 Measurement"]
N3["47.3 Issues"]
N4["47.4 Lessons"]
N5["Planning the trial safely"]
N6["Recording the timeline"]
N7["Extending the trial programme"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX139"]
[Learning outcomes]
|
v
[47.1 Trial]
|
v
[47.2 Measurement]
|
v
[47.3 Issues]
|
v
[47.4 Lessons]
|
v
[Planning the trial safely]
|
v
[Recording the timeline]
|
v
[Extending the trial programme]
|
v
[Worked example] --> [Exercise EX139]
Learning outcomes
You will (1) run a live recovery trial; (2) measure recovery; (3) learn.
47.1 Trial
To prove the recovery plan works outside the lab, Kofi and the council engineers carry out a live trial: with change board approval, they replace a working branch firewall with the spare during a weekend window when the office is closed. The trial follows the recovery runbook exactly, as if the original firewall had failed, and a second engineer watches for any step that is unclear or missing. Running a recovery for real, rather than discussing it, is the only reliable way to know how long it takes.
47.2 Measurement
The target, set in the resilience design, is to restore service at a branch within four hours of deciding to replace its firewall. Timing starts when the old unit is disconnected and stops when the regression tests for that office pass on the new one. The trial takes two hours and fifty minutes, comfortably inside the target. Kofi also records how long the office would have been without central services, and confirms that the office's local fallback arrangements would have covered that period.
47.3 Issues
The trial finds one significant issue: the spare firewall was running firmware two versions older than production, and updating it added thirty minutes. The quarterly update of the spare had been missed during a busy period. In a real outage, those thirty minutes might matter. The team adds the spare's firmware check to the monthly routine checklist, not just the quarterly one, and adds a step to the runbook to check the firmware version first. Finding this during a planned trial is far better than finding it during a real failure.
47.4 Lessons
After the trial, the team holds a short review. The runbook is updated with the firmware check, a clearer step for loading the per-site configuration and a realistic time estimate for each stage. The trial report records the timeline, the issue and the changes made, and goes into the dossier as evidence for the recovery requirement. The council agrees to repeat a recovery trial each year, rotating among engineers, so that recovery skills and spare equipment both stay ready rather than decaying between rare real failures.
Planning the trial safely
A live trial changes a working office, so it needs careful planning. Kofi writes a plan with the scope, the window, the success measure, the rollback step and the people involved. The office manager is informed, and the trial is scheduled for a weekend when the office is closed. The original firewall is kept powered off but connected and ready, so rollback takes minutes. The council's change advisory board approves the plan. A stop condition is defined in advance: if the replacement is not passing traffic within three hours, the original is restored and the trial is recorded as incomplete.
Recording the timeline
During the trial, one engineer acts as recorder and notes the time of each step: start, old unit removed, spare connected, configuration loaded, firmware updated, tunnels established, tests passed and finish. The timeline shows exactly where time went. Loading the configuration from backup took eight minutes; the firmware update took thirty; tunnel establishment took four. Without the timeline, the team would only know the total and might guess wrongly about which step to improve. The recorded timeline becomes part of the updated runbook as a realistic estimate for future replacements at any branch office.
Extending the trial programme
One successful trial does not prove every recovery path. Kofi proposes an annual programme: replacing a branch firewall from spare, restoring the main firewall pair configuration from backup in the lab, recovering the jump hosts and rebuilding a VPN tunnel with a new certificate. Each trial rotates among engineers so that recovery knowledge is not held by one person. Results are reviewed against targets, and runbooks are updated after each trial. Regular trials also keep spares, backups and certificates in a known good state, since problems such as outdated firmware are found before a real failure rather than during one.
Worked example
The trial report is filed.
Synthetic case. The trial is synthetic.
Errors and limits
Recovery plans untested in reality often miss small but time-consuming steps.
Exercises
- EX139 — Trial time. How long did recovery take? Rubric: 3 = 2h50 within 4h;; 2 = under 4h;; 1 = vague;; 0 = none.
- EX140 — Firmware issue. What issue slowed recovery? Rubric: 3 = old firmware on spare;; 2 = firmware;; 1 = vague;; 0 = none.
- EX141 — Approval. Why was approval needed? Rubric: 3 = live service change;; 2 = policy;; 1 = vague;; 0 = none.
Synthesis
A live trial proves recovery works and improves the runbook.
Continue studying — chapter 47 of 50
← Chapter 46 (draft) · Course chapter index · Chapter 48 (draft) →