Chapter 47 — Failure and recovery trial

Lesson map

Lesson 47 published. Map, example, EX139.

flowchart TD
  N0["Learning outcomes"]
  N1["47.1 Trial"]
  N2["47.2 Detection"]
  N3["47.3 Recovery"]
  N4["47.4 Lessons"]
  N5["47.5 Improvement"]
  N6["Trial safety"]
  N7["Measurements"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX139"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["47.1 Trial"]
  N2["47.2 Detection"]
  N3["47.3 Recovery"]
  N4["47.4 Lessons"]
  N5["47.5 Improvement"]
  N6["Trial safety"]
  N7["Measurements"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX139"]

[Learning outcomes]
    |
    v
[47.1 Trial]
    |
    v
[47.2 Detection]
    |
    v
[47.3 Recovery]
    |
    v
[47.4 Lessons]
    |
    v
[47.5 Improvement]
    |
    v
[Trial safety]
    |
    v
[Measurements]
    |
    v
[Worked example] --> [Exercise EX139]

Learning outcomes

You will (1) run a full failure and recovery trial; (2) measure recovery; (3) learn lessons.

47.1 Trial

With Fernhill's agreement, a branch simulates a bad image release after closing time.

The simulated bad release was a test image signed with a lab key and approved by the change board only for this branch and this night. Real patrons were not involved, since the branch was closed. Marta, the technician and Mateo were present, with the stop condition agreed in advance. The aim was to see whether detection, rollback and the people following the runbooks would work together as designed, not just in separate tests.

47.2 Detection

Restore digests fail; the five-failure alert fires within minutes.

The digest check after restore found that the policy files did not match the manifest of the approved release, so each PC reported failure. Because many PCs failed in the same night, the incident alert fired as the probability work had predicted for a shared cause. The time from the first failure to the alert was recorded. The technician then started the rollback runbook without waiting for further instructions. Every step was timed.

47.3 Recovery

Rollback to the previous image completes for the branch in 40 minutes.

Forty minutes covered the whole branch, with PCs rolling back in groups to avoid loading the network. Each PC verified the previous image's digests after rollback and reported success to the dashboard. No PC needed the manual restore described in the stop condition. The figure is longer than the ten minutes measured for a single lab PC, as expected for many PCs sharing one link. The dashboard showed every PC healthy before midnight.

47.4 Lessons

The runbook lacked a step for confirming printer mappings after rollback.

After rollback the printers worked on most PCs, but the runbook did not tell the technician to check them, and two PCs had lost their mapping. On a busy morning, patrons would have discovered the problem first. The gap was invisible in the technical tests because they checked the image, not the full patron journey. The debrief recorded it as the trial's main finding. Nobody had noticed it before the trial.

47.5 Improvement

The step is added and tested.

The new step tells the technician to print a test page from one PC in each room after rollback. Mateo tested the revised runbook on lab PCs with a synthetic rollback, then the technician followed it on her own. The change is recorded with the trial report that prompted it, and the runbook's date and version are updated. The next annual trial will check the step again under realistic conditions. Marta approved the revision.

Trial safety

The trial is planned with clear limits: one branch, after closing, lab-signed test image, with rollback ready and Marta present. A stop condition is defined: if rollback fails for any PC, the technician restores it manually from the local copy. Patrons are not affected.

The plan is approved by Fernhill's change board.

Measurements

The team records detection time, rollback time and any manual steps needed.

Debrief

A debrief the next day lists what worked and what to improve.

Repeat

The trial will be repeated yearly.

Communication

Branch staff are told the next morning that a trial took place and that PCs were ready as usual.

Evidence

Trial records join the dossier.

Comparison

Results match the design's expectations except the printer step.

Confidence

The trial increases confidence in rollback.

Approval

The change board accepts the results.

Why trials matter

A trial tests people and procedures as well as technology. The runbook gap would not have appeared in a technical test alone; it needed someone following the steps under realistic conditions.

Improvement tracking

The added printer step is tested on the next release and works as expected.

Recovery check

Every branch PC is ready by morning.

Worked example

Marta notes that without the trial, the printer issue would have appeared on a busy morning.

Synthetic case. Synthetic.

Errors and limits

Trials outside opening hours protect patrons while testing realistically.

Exercises

• EX139 — Detection. How was the bad image detected?
  Show marking criteria

  Rubric: 3 = digest failures, five-failure alert; 2 = partial; 1 = vague; 0 = none.
• EX140 — Recovery time. How long did branch rollback take?
  Show marking criteria

  Rubric: 3 = 40 minutes; 2 = partial; 1 = vague; 0 = none.
• EX141 — Lesson. What did the trial reveal?
  Show marking criteria

  Rubric: 3 = missing printer step; 2 = partial; 1 = vague; 0 = none.

Synthesis

Rehearse failure to find gaps before real incidents.

Continue studying — chapter 47 of 50

← Chapter 46 (draft) · Course chapter index · Chapter 48 (draft) →