Chapter 47 — Failure and recovery trial

Lesson map

Lesson 47 published. Map, example, EX139.

flowchart TD
  N0["Learning outcomes"]
  N1["47.1 Scenario"]
  N2["47.2 Performance"]
  N3["47.3 Human factors"]
  N4["47.4 Lessons"]
  N0 --> N1 --> N2 --> N3 --> N4
  N4 --> EX["Worked example"]
  EX --> Q["Exercise EX139"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["47.1 Scenario"]
  N2["47.2 Performance"]
  N3["47.3 Human factors"]
  N4["47.4 Lessons"]
  N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX139"]

[Learning outcomes]
    |
    v
[47.1 Scenario]
    |
    v
[47.2 Performance]
    |
    v
[47.3 Human factors]
    |
    v
[47.4 Lessons]
    |
    v
[Worked example] --> [Exercise EX139]

Learning outcomes

You will (1) run a realistic failure exercise; (2) observe human and technical performance; (3) produce lessons for improvement.

47.1 Scenario

With approval from Saltmarsh's directors, Lindiwe runs a full recovery trial in the isolated environment, treating it as a live incident. The scenario: at six on a Monday morning, ransomware has encrypted dispatch servers and the nightly backup share. Only immutable backups remain. The incident team does not know the scenario in advance. Prof. Hale and the security lead observe, recording decisions and times without intervening. Lindiwe prepares the scenario materials, a timeline of injected events and an observer sheet for recording what happens.

47.2 Performance

The team follows the incident process: detection, decision to recover, authorisation by the named directors and execution of the runbook. The rebuild completes in three hours and thirty-five minutes. Two problems occur. The on-call manager cannot find the offline credentials at first, losing twenty minutes. The restored data are verified with checksums, and one batch fails verification, so the team restores it from the previous snapshot. The team records every action in the incident log, which Lindiwe later uses to reconstruct the timeline accurately.

47.3 Human factors

Lindiwe observes that stress affected communication: two people ran the same script simultaneously, and depot supervisors were not told when to switch to printed lists. She notes these are process issues, not individual failings. She recommends a single coordinator role during recovery and a message template for depots. Prof. Hale notes that recovery trials test people and communication as much as technology. She discusses her observations with the participants in a short debrief, and they agree with the proposed changes and add one of their own about shift handovers.

47.4 Lessons

The lessons report lists five improvements: label and check offline credential storage monthly, add a coordinator role, use the depot message template, automate checksum verification of restored batches and repeat the trial annually with different staff. Each has an owner and date. Lindiwe notes that the trial met the objective despite problems, which gives confidence while showing exactly where the margin could be lost. The directors approve the improvements and ask for a short progress update in three months, which Lindiwe schedules before her course ends.

Worked example

Lindiwe runs an unannounced recovery trial completed in 3 hours 35 minutes, observes credential and communication problems and records five owned improvements.

Synthetic case. The trial is synthetic.

Errors and limits

Recovery trials that test only technology miss communication failures.

Exercises

• EX139 — Trial duration. How long did the rebuild take?
  Show marking criteria

  Rubric: 3 = three hours thirty-five minutes;; 2 = partly correct;; 1 = wrong;; 0 = none.
• EX140 — Credential delay. How much time was lost finding credentials?
  Show marking criteria

  Rubric: 3 = twenty minutes;; 2 = partly correct;; 1 = wrong;; 0 = none.
• EX141 — Coordinator role. Why add a coordinator role?
  Show marking criteria

  Rubric: 3 = two people ran the same script;; 2 = partly correct;; 1 = vague;; 0 = none.