Chapter 47 — Failure and recovery trial
Lesson map
Lesson 47 published. Map, example, EX139.
flowchart TD
N0["Learning outcomes"]
N1["47.1 Planning the trial"]
N2["47.2 Running and measuring"]
N3["Lab — Recovery trial timings"]
N4["47.3 Improving the procedure"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX139"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["47.1 Planning the trial"]
N2["47.2 Running and measuring"]
N3["Lab — Recovery trial timings"]
N4["47.3 Improving the procedure"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX139"]
[Learning outcomes]
|
v
[47.1 Planning the trial]
|
v
[47.2 Running and measuring]
|
v
[Lab — Recovery trial timings]
|
v
[47.3 Improving the procedure]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX139]
Learning outcomes
You will (1) plan a controlled failure and recovery trial; (2) measure recovery against objectives; (3) improve recovery procedures from trial evidence.
47.1 Planning the trial
Sofia's capstone includes a recovery trial in the lab mirror. The scenario: ransomware encrypts the fare-calculation servers, and the team must restore service from offline backups while validators continue offline. Objectives are a recovery time of four hours for fare calculation and no loss of tap records, since validators buffer them. She writes the plan with roles, steps, success criteria, safety limits and an abort procedure, and informs everyone involved. Trials are rehearsed carefully because the purpose is to learn safely, not to cause a real outage.
47.2 Running and measuring
During the trial, the team rebuilds servers from images, restores the database from the last offline backup, reloads verified tariff tables and replays buffered tap records. Sofia records times for each step. The lab below summarises the synthetic timings: total recovery exceeds the four-hour objective because database restore and record replay take longer than planned. No tap records are lost. Measured times replace optimistic estimates in the recovery plan, which is the main value of the trial: the plan now reflects what the team can actually do.
Lab — Recovery trial timings
# Synthetic recovery trial step timings (minutes) vs a 4-hour objective.
steps = [("declare and isolate", 20), ("rebuild servers from images", 55), ("restore database", 120),
("reload verified tariffs", 15), ("replay buffered tap records", 95), ("reconcile and sign-off", 20)]
total = sum(m for _, m in steps)
for s, m in steps: print(f"{s:30} {m:4d} min")
print(f"total {total} min = {total/60:.1f} h; objective 240 min; met: {total <= 240}")
print("tap records lost: 0 (validators buffered all records)")
Run with python3 lab_recovery.py. Verified output:
declare and isolate 20 min
rebuild servers from images 55 min
restore database 120 min
reload verified tariffs 15 min
replay buffered tap records 95 min
reconcile and sign-off 20 min
total 325 min = 5.4 h; objective 240 min; met: False
tap records lost: 0 (validators buffered all records)
47.3 Improving the procedure
The team analyses the overrun. Database restore was slow because backups were compressed with a setting that limits parallel restore; record replay was slow because it processed records one at a time. Sofia proposes changing the backup format and batching replay, then reruns those two steps, cutting their combined time sharply. She updates the recovery runbook and schedules the next full trial. NIST CSF 2.0's recover function emphasises improving recovery planning from lessons learned, and the trial puts that principle into practice with numbers rather than opinions.
Worked example
Sofia runs a ransomware recovery trial in the lab mirror, measures a total time over the four-hour objective with no record loss, and fixes restore and replay bottlenecks.
Synthetic case. The trial, timings and fixes are synthetic; NIST CSF 2.0 is real.
Errors and limits
Lab trials omit real-world stress and coordination problems. Backups must be tested for integrity, not only speed.
Exercises
- EX139 — Trial objectives. What were the two recovery objectives? Rubric: 3 = fare calculation restored in four hours and no tap records lost;; 2 = both;; 1 = one;; 0 = none.
- EX140 — Lab timing result. What total recovery time did the lab report, and which steps overran? Rubric: 3 = 325 minutes (5.4 hours); database restore (120 min) and record replay (95 min) dominated;; 2 = both;; 1 = met;; 0 = none.
- EX141 — Two fixes. Which two fixes did Sofia propose? Rubric: 3 = change backup compression to allow parallel restore; batch record replay;; 2 = both;; 1 = one;; 0 = none.
Synthesis
Recovery trials replace optimistic estimates with measured times and drive concrete improvements to recovery procedures.
Continue studying — chapter 47 of 50
← Chapter 46 (draft) · Course chapter index · Chapter 48 (draft) →