Chapter 47 — Failure and recovery trial

Lesson map

Lesson 47 published. Map, example, EX139.

flowchart TD
  N0["Learning outcomes"]
  N1["47.1 Designing the trial"]
  N2["47.2 Observations"]
  N3["47.3 Improvements"]
  N4["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4
  N4 --> EX["Worked example"]
  EX --> Q["Exercise EX139"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["47.1 Designing the trial"]
  N2["47.2 Observations"]
  N3["47.3 Improvements"]
  N4["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX139"]

[Learning outcomes]
    |
    v
[47.1 Designing the trial]
    |
    v
[47.2 Observations]
    |
    v
[47.3 Improvements]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX139]

Learning outcomes

You will (1) design a failure trial for a GRC service; (2) observe detection and recovery; (3) improve the service from the results.

47.1 Designing the trial

With the security manager, Nadia plans a trial of how the monitoring service behaves under failure. Three failures are injected in a controlled window, with owners informed in advance: a collector script crashes, a source system becomes unavailable and an evidence file is altered after storage. The trial tests whether each failure is detected, how quickly and whether recovery restores correct evidence. The rules exclude any action on production control systems or customer data. Announcing the trial to owners prevents an injected failure being treated as a real incident.

47.2 Observations

The crashed collector is detected at the next scheduled run and alerts the owner within a day. The unavailable source raises a missing-evidence alert as designed. The altered file is detected by a hash mismatch at the weekly integrity check, but not sooner. Recovery from the crash takes two hours because the runbook omits one step. Nadia records each observation with timestamps and compares it with expectations, noting that detection of tampering is slower than the team assumed. Prof. Brenner asks how long an altered file could have gone unnoticed, and the answer, up to a week, motivates the daily check.

47.3 Improvements

The team adds the missing runbook step and moves integrity checks from weekly to daily. They also add an alert when integrity checks themselves fail to run. Nadia notes that the trial revealed weaknesses that would have remained hidden in normal operation. She learns that planned failure trials are the governance equivalent of fire drills: they prove whether detection and recovery work, and they are only safe and useful when carefully scoped and announced. Alerting when an integrity check fails to run closes the same silence-as-success gap she fixed during the design review.

Worked example

Nadia runs three announced failure injections, finds slow tamper detection and a runbook gap and fixes both.

Synthetic case. The trial is synthetic and excluded production control systems.

Errors and limits

Announced trials differ from real failures. Daily checks add load.

Exercises

• EX139 — Injected failures. Which three failures were injected?
  Show marking criteria

  Rubric: 3 = collector crash, source unavailable, altered evidence file;; 2 = three;; 1 = vague;; 0 = none.
• EX140 — Slow detection. Which failure was detected slowly?
  Show marking criteria

  Rubric: 3 = the altered evidence file, at the weekly check;; 2 = tampering;; 1 = vague;; 0 = none.
• EX141 — Check change. How often do integrity checks run after the trial?
  Show marking criteria

  Rubric: 3 = daily;; 2 = daily, hedged with weekly;; 1 = weekly;; 0 = none.

Synthesis

Announced failure trials prove detection and recovery and reveal hidden weaknesses.

Continue studying — chapter 47 of 50

← Chapter 46 (draft) · Course chapter index · Chapter 48 (draft) →