Chapter 47 — Failure and recovery trial
Lesson map
Lesson 47 published. Map, example, EX139.
flowchart TD
N0["Learning outcomes"]
N1["47.1 Purpose"]
N2["47.2 Scenario"]
N3["47.3 Objectives"]
N4["47.4 Execution"]
N5["47.5 Fix during trial"]
N6["47.6 Debrief"]
N7["47.7 Recording"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX139"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["47.1 Purpose"]
N2["47.2 Scenario"]
N3["47.3 Objectives"]
N4["47.4 Execution"]
N5["47.5 Fix during trial"]
N6["47.6 Debrief"]
N7["47.7 Recording"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX139"]
[Learning outcomes]
|
v
[47.1 Purpose]
|
v
[47.2 Scenario]
|
v
[47.3 Objectives]
|
v
[47.4 Execution]
|
v
[47.5 Fix during trial]
|
v
[47.6 Debrief]
|
v
[47.7 Recording]
|
v
[Worked example] --> [Exercise EX139]
Learning outcomes
You will (1) design a controlled failure trial; (2) execute recovery under observation; (3) measure recovery against objectives; (4) improve procedures from results.
47.1 Purpose
Restore tests check backups; a failure trial checks the whole organisation's response: people, procedures and systems together.
A restore test can pass while a real outage still goes badly: the person who knows the procedure is on leave, the spare hardware is missing a part, or nobody tells the courts about delays. A trial exposes those gaps by making the people who would actually respond do the work under realistic conditions. It is planned in advance so that no live evidence is put at risk, yet it is treated seriously enough to show genuine weaknesses.
47.2 Scenario
On a planned Saturday, the EvidenceDesk server is declared lost (powered off and treated as destroyed). The IT officer, with Priya observing only, must restore service on a spare machine, verify all records against anchors and resume intake.
Treating the server as destroyed means nothing may be copied from it, not even a quick configuration file, because a fire or theft would allow no such shortcut. Choosing a Saturday keeps casework unaffected. Priya observes without helping, because her presence as the system's author would hide whether the documentation alone is sufficient. She records each step's start and end time and every point where the IT officer hesitates or consults her notes.
47.3 Objectives
Recovery within four hours; zero record loss; anchors verify; examiners can resume work.
Four hours matches the restore target set earlier for reliability, so the trial tests a real commitment rather than an arbitrary figure. Zero record loss is checked by comparing the restored chain's length and final link with the last anchor. Resuming work means a test examiner can open a record, verify a working copy and add an entry, which shows the whole flow, not just the database, is back in service.
47.4 Execution
The IT officer follows the procedure. Restore from the latest write-once backup takes 40 minutes; chain verification 3 minutes. The last anchor matches. One step fails: the spare machine's clock is 9 minutes slow, which would stamp new entries wrongly.
Forty minutes for the restore and three for chain verification leave ample margin against the four-hour target. The clock problem is the more instructive result: nothing in the restore procedure checked time, and a slow clock would have produced custody entries that appear to come before events they actually followed. That kind of ordering error is exactly what a defence expert would look for, so it matters even though no data was lost.
47.5 Fix during trial
The IT officer syncs time before resuming intake, as the checklist should have required. Total time to resume: 2 hours 10 minutes.
Fixing the clock on the spot was right because the objective was to resume service correctly, and the trial would otherwise end with a known fault in place. The IT officer records the drift found, the source used for synchronisation and the time the correction was made. Two hours ten minutes is well within target, and the trial report notes that roughly half that time went to unplanned setup work on the spare machine.
47.6 Debrief
What went well, what did not, actions. Action: add a time-sync check to the recovery checklist and to EvidenceDesk startup (refuse to write if drift exceeds one minute).
The debrief happens the same afternoon while memories are fresh, with the IT officer, the quality manager and Priya. Each action has an owner and a date. Putting the drift check into EvidenceDesk itself, not just the checklist, means the safeguard works even if a future administrator skips a step. The one-minute threshold is chosen because entries are recorded to the minute, so smaller drift would not change their order.
47.7 Recording
The trial report records timings, deviations and actions, signed by the IT officer and quality manager.
Timings show whether the objectives were met; deviations show where reality departed from the written procedure; actions show what will change. Two signatures confirm that both the operator and the quality function accept the account as accurate. The report is filed with the reliability records, so the next trial can compare its own timings and check that every earlier action was actually completed. Copies go to Hugo.
47.8 Frequency
Annual failure trials, plus after major changes.
An annual trial keeps the skill alive as staff change and memory fades. Major changes, such as new server hardware, a new storage platform or a change in backup media, trigger an extra trial because they alter exactly what a recovery depends on. The quality manager keeps the schedule and makes sure each trial is run by someone who did not run the previous one, so knowledge is spread across the team. Dates are fixed at the start of each year.
Observer role
Priya observes without helping, timing each step and noting hesitations. Her silence is deliberate: if the author fixes problems during the trial, it shows only that the author can recover the system, not that the lab can. Her notes feed the debrief alongside the IT officer's account.
Worked example
The startup drift check is implemented: on the spare machine with a slow clock, EvidenceDesk now refuses writes and displays the drift. After sync, it starts normally. A regression test simulates drift.
Synthetic case. The trial is invented.
Errors and limits
A planned trial lacks the stress of a real failure; vary scenarios and timing.
Exercises
- EX139 — Trial vs restore test. How does a failure trial differ from a restore test? Rubric: 3 = tests people, procedures and systems together; 2 = partial; 1 = same; 0 = none.
- EX140 — Clock issue. What problem appeared, and why does it matter? Rubric: 3 = spare clock 9 min slow; wrong timestamps on entries; 2 = partial; 1 = vague; 0 = none.
- EX141 — Startup check. What did the debrief add to EvidenceDesk? Rubric: 3 = refuse writes if drift >1 min; 2 = partial; 1 = vague; 0 = none.
Synthesis
Trials of whole-system recovery reveal gaps that component tests miss; feed every gap back into tools and checklists.
Continue studying — chapter 47 of 50
← Chapter 46 (draft) · Course chapter index · Chapter 48 (draft) →