◇ OpenCyber Academy
Engineering · Chapter 47 · English

Chapter 47 — Failure and recovery trial

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 47 — Failure and recovery trial

Lesson map

Lesson 47 published. Map, example, EX139.

flowchart TD
  N0["Learning outcomes"]
  N1["47.1 Planning the trial"]
  N2["47.2 Running the trial"]
  N3["Lab — Recovery times against targets for three failu"]
  N4["47.3 Learning from failure"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX139"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["47.1 Planning the trial"]
  N2["47.2 Running the trial"]
  N3["Lab — Recovery times against targets for three failu"]
  N4["47.3 Learning from failure"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX139"]

[Learning outcomes]
    |
    v
[47.1 Planning the trial]
    |
    v
[47.2 Running the trial]
    |
    v
[Lab — Recovery times against targets for three failu]
    |
    v
[47.3 Learning from failure]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX139]

Learning outcomes

You will (1) plan a failure trial for the reporting and verification system; (2) run it safely; (3) learn from what failed.

47.1 Planning the trial

Aiko plans a trial to test how the programme behaves when parts fail. Scenarios include the triage tool going offline during a campaign, the second finance officer being absent at month-end and the security team being unreachable at night. The trial is announced to all participants, scheduled at a quiet time and approved by operations and safety managers. No staff are deceived; the trial tests systems and procedures, not individuals. Each scenario has a defined start, observer and stop condition to ensure safety.

47.2 Running the trial

In the first scenario, the triage tool is taken offline for an hour. Reports queue in the mailbox as designed, but staff receive no thank-you messages and some report twice. In the second, the finance hold escalates correctly to a backup officer. In the third, the control room reaches the out-of-hours security contact within ten minutes using the rota. The lab summarises the recovery time for each scenario against its target, showing that only the triage outage exceeded its target. Observers time each step.

Lab — Recovery times against targets for three failure scenarios

# Recovery times against targets (minutes, synthetic).
s = [("triage outage", 60, 30), ("finance backup", 15, 60), ("out-of-hours contact", 10, 15)]
for name, t, target in s: print(f"{name}: {t} min vs target {target} -> {'MISSED' if t > target else 'met'}")

Run with python3 lab_recovery.py. Verified output:

triage outage: 60 min vs target 30 -> MISSED
finance backup: 15 min vs target 60 -> met
out-of-hours contact: 10 min vs target 15 -> met

47.3 Learning from failure

Aiko holds a blameless review. The triage outage revealed that the fallback had no automatic acknowledgement, so staff were uncertain whether reports had arrived. The team adds a simple automatic reply from the mailbox itself, independent of the triage tool. Aiko notes that failures found in planned trials are much cheaper than those found during real attacks. She updates the verification table with a new requirement for acknowledgement during outages, closing the loop between trial and design. The reply is tested monthly.

Worked example

Aiko runs an announced failure trial of triage, finance and out-of-hours scenarios, finds a missing acknowledgement during outages and adds a requirement.

Synthetic case. The trial is synthetic.

Errors and limits

Planned trials are less stressful than real failures. Quiet-time trials miss peak effects.

Exercises

Synthesis

Announced failure trials find weaknesses cheaply and feed new requirements back into design.

Continue studying — chapter 47 of 50

← Chapter 46 (draft) · Course chapter index · Chapter 48 (draft) →