Chapter 47 — Failure and recovery trial
Lesson map
Lesson 47 published. Map, example, EX093.
flowchart TD
N0["Learning outcomes and connection"]
N1["47.1 Why trial recovery"]
N2["47.2 Designing the trial"]
N3["47.3 Critical path"]
N4["47.5 Practical activity: Lab 17"]
N5["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX093"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["47.1 Why trial recovery"]
N2["47.2 Designing the trial"]
N3["47.3 Critical path"]
N4["47.5 Practical activity: Lab 17"]
N5["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX093"]
[Learning outcomes and connection]
|
v
[47.1 Why trial recovery]
|
v
[47.2 Designing the trial]
|
v
[47.3 Critical path]
|
v
[47.5 Practical activity: Lab 17]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX093]
Learning outcomes and connection
After this chapter you can (1) design a controlled failure and recovery trial with safety boundaries; (2) compute critical paths and compare planned and measured recovery times; (3) observe and record human and technical performance during the trial; and (4) turn trial results into requirement status and design changes. It validates recovery requirements (R6) and prepares the evidence dossier (Chapter 48).
47.1 Why trial recovery
Recovery capability that has not been exercised is a hope. Backups that have not been restored may be incomplete; procedures that have not been followed may be wrong; times that have not been measured are guesses. NIST SP 800-34 Rev. 1 covers contingency planning, including testing, training and exercises.
TMRP's own trial showed why: the eight-hour restoration target in R6 looked reasonable until the critical path was measured. Measuring it was the point.
47.2 Designing the trial
Define the scenario (e.g., SCADA servers declared compromised at 02:00), the objectives (restore remote operation from offline backups; maintain supply in local mode throughout), the safety boundaries (trial on the test environment or a planned window with local operation staffed; abort criteria), the observers and the measurements (time of each task, deviations from procedure, decisions).
TMRP's trial used the scenario of SCADA servers declared compromised, with supply maintained in local mode throughout. Observers recorded each task's time and every deviation from the procedure.
47.3 Critical path
Recovery tasks have durations and dependencies. The critical path is the longest chain of dependent tasks; it sets the minimum recovery time. Tasks off the critical path have slack. Shortening recovery means shortening critical-path tasks or breaking dependencies — not speeding up tasks with slack.
In TMRP's trial, re-keying the RTUs had six hours of slack. Speeding it up would have changed nothing. Only imaging, rebuild and configuration restore set the twelve-hour result, which is why every option for meeting R6 targets one of those three tasks, such as pre-built standby images or imaging on spare hardware.
47.4 Worked example: TMRP recovery trial (synthetic)
The trial plan (Lab 17): isolate (1 h); manual operations established (1 h); forensic imaging of the SCADA servers (3 h); rebuild SCADA from offline media (4 h); restore configuration (2 h); re-key RTUs (3 h, in parallel); verify setpoints (1 h); resume remote control (1 h). The critical path runs through forensic imaging, rebuild and configuration restore: remote control returns at 12 h, against R6's 8 h. Re-keying has 6 h of slack — speeding it up achieves nothing. Options to meet 8 h: pre-built standby SCADA images (rebuild 4 h → 1 h); imaging in parallel with rebuild on spare hardware (removes 3 h from the path); or agreement with stakeholders that R6 be 12 h because supply is maintained in local mode throughout. The trial also measured that manual operations were established in 50 minutes, meeting the 1 h requirement.
47.5 Practical activity: Lab 17
# Lab 17 - recovery critical path versus the restoration target (synthetic TMRP trial plan, offline)
T = { # task: (hours, prerequisites)
"isolate": (1, []), "manual_ops": (1, ["isolate"]), "forensic_image": (3, ["isolate"]),
"rebuild_scada": (4, ["forensic_image"]), "restore_config": (2, ["rebuild_scada"]),
"rekey_rtus": (3, ["isolate"]), "verify_setpoints": (1, ["restore_config", "rekey_rtus"]),
"resume_remote": (1, ["verify_setpoints"])}
ef, prev = {}, {}
def finish(t):
if t not in ef:
d, ps = T[t]; start = max((finish(p) for p in ps), default=0)
prev[t] = max(ps, key=finish) if ps else None; ef[t] = start + d
return ef[t]
end = finish("resume_remote"); path, n = [], "resume_remote"
while n: path.append(n); n = prev[n]
print("restore remote control at", end, "h; critical path:", " -> ".join(reversed(path)))
print("slack of rekey_rtus:", finish("restore_config") - finish("rekey_rtus"), "h; target 8 h met:", end <= 8)
Expected output:
restore remote control at 12 h; critical path: isolate -> forensic_image -> rebuild_scada -> restore_config -> verify_setpoints -> resume_remote
slack of rekey_rtus: 6 h; target 8 h met: False
The output confirms the worked example: remote control returns after twelve hours because imaging, rebuild and configuration restore run in series, and re-keying finishes six hours before it is needed. Change the plan as EX093 asks and watch the critical path move to a different chain. The lab does not include manual operations on the path to remote control, because they run in parallel; the trial measured them separately at fifty minutes against the one-hour requirement. Keep both outputs.
47.6 Common errors and limits
- Recovery tested only on paper.
- Trials without abort criteria or safety staffing.
- Improving tasks that have slack.
- Trial results reported as passes when the requirement was missed.
- Forensic preservation skipped to save time without a decision.
TMRP's trial avoided the fourth error by reporting R6 as not met at twelve hours, and it kept forensic imaging rather than skipping it without a decision. Abort criteria and local staffing were set before the trial began, and the options for meeting R6 target only critical-path tasks.
Exercises — chapter 47
EX093 — Shorten the path. Objective: use the critical path. Inputs: Lab 17. Steps: (1) make rebuild_scada depend only on isolate (spare hardware, imaging in parallel) and recompute; (2) additionally reduce rebuild_scada to 1 h (standby image) and recompute; report the new critical path each time. Expected evidence: outputs. Rubric: 3 = correct results (run code) and the observation that the critical path moves to another chain; 2 = numbers right, path not discussed; 1 = not run; 0 = none.
EX094 — Forensics vs speed. Objective: make a documented trade-off. Inputs: skipping forensic imaging shortens recovery. Steps: write a decision note for the operations director: options, consequences for investigation, legal/regulatory and future prevention, recommendation. Expected evidence: one page. Rubric: 3 = parallel imaging on spare hardware as preferred, risks of losing evidence explained, decision owner; 2 = recommends skipping without consequences; 1 = vague; 0 = none.
Chapter synthesis
Recovery must be trialled to be trusted. Design trials with safety boundaries and measurements, find the critical path, improve only what is on it, and report the requirement status the trial actually shows — then change the design or the requirement with stakeholders.
Continue studying — chapter 47 of 50
← Chapter 46 · Course chapter index · Chapter 48 →
Continue to the next authored lesson.