Chapter 47 — Failure and recovery trial with reproducible data
Lesson map
Lesson 47 published. Map, example, EX093.
flowchart TD
N0["Learning outcomes and connection"]
N1["47.1 From fault injection to recovery trial"]
N2["47.2 Reproducible inputs"]
N3["47.4 Comparing with objectives"]
N4["47.5 Reproducibility check"]
N5["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX093"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["47.1 From fault injection to recovery trial"]
N2["47.2 Reproducible inputs"]
N3["47.4 Comparing with objectives"]
N4["47.5 Reproducibility check"]
N5["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX093"]
[Learning outcomes and connection]
|
v
[47.1 From fault injection to recovery trial]
|
v
[47.2 Reproducible inputs]
|
v
[47.4 Comparing with objectives]
|
v
[47.5 Reproducibility check]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX093]
Learning outcomes and connection
After this chapter you can (1) design a recovery trial around a recorded failure mode with seed data and restore points; (2) publish trial inputs so others can reproduce the result; (3) measure repeatability across runs; and (4) compare results with recovery objectives and state residual risk. It builds on Chapters 13, 35, 40 and 46 and prepares Chapters 48–50.
47.1 From fault injection to recovery trial
Chapter 40 injected faults to test error handling. A recovery trial tests the whole recovery claim: after failure F during activity A, the system returns to state S within time T with data loss at most L, and evidence integrity holds. It is run with recorded inputs so that the result can be reproduced and compared across design versions.
In the worked example below, F is a corrupted evidence-store volume at minute 25 and A is case C-9 receiving 600 items. T is 30 minutes and L is zero items.
47.2 Reproducible inputs
Publish (within the course or organisation): seed data generator and seed; the event sequence (or generator parameters); restore points (backup snapshots with hashes); the failure script (what fails, when, how); environment definition (container images or setup script with pinned versions); and the analysis script. Another person running the package should obtain the same measurements, within stated tolerance for timing.
For the C-9 trial, the package would contain the item generator with seeds 1–10, the minute-0 snapshot with its hash and the script that deletes the store's data directory at minute 25. It would also include the analysis script that computes median and maximum recovery time and counts lost items per seed.
47.3 Worked example (synthetic): evidence-store failure during an active case
Scenario: the capstone system is handling case C-9 with 600 evidence items arriving over 60 minutes; at minute 25 the evidence-store volume is corrupted (simulated by deleting the store's data directory and marking the service unhealthy). Recovery: restore the store from the snapshot taken at minute 0 plus replay of the outbox spill and the raw-event archive; verify the manifest; resume.
Objectives: store usable within 30 minutes of failure; zero evidence items lost; manifest verifies after recovery; case workflow uninterrupted (approvals continue against queued evidence references).
Results over 10 runs (seeds 1–10): recovery time median 18 min, max 24 min — objective met; items lost: 0 in 9 runs, 3 in one run (seed 7), where items arrived during the 40-second window between the store being marked unhealthy and the outbox switching on — a race in health detection; manifest verified in all runs after recovery, but in seed 7 the anchor flagged 3 missing items correctly (tamper-evidence worked as designed, reporting the loss). Workflow continued in all runs.
Repeatability: recovery time standard deviation 3.1 min; the loss occurred in 1 of 10 runs, so a single run would likely have missed it. Fix: writes go through the outbox always (write-behind), not only when the store is unhealthy; rerun of all 10 seeds: zero loss, median recovery 19 min.
Residual risk stated: snapshot at minute 0 means replay depends on raw archive integrity; raw archive shares the storage cluster with the evidence store (common cause) — owner: platform lead; action: move raw archive to a separate cluster; until then, claim qualified.
47.4 Comparing with objectives
Report a table: objective, measured value (median, worst, count of failures), met / not met, notes. "Met in 9 of 10 runs" is not "met"; it is a defect to fix or a limit to state.
For the C-9 trial before the fix, that table would show recovery time with a median of 18 min and a worst case of 24 min, met. Items lost would read 0 in 9 runs and 3 in seed 7, not met, which is why the write-behind change was required before the claim could stand.
47.5 Reproducibility check
Ask a peer to run the published package. Differences reveal hidden dependencies — an unpinned library, a local configuration file, timing assumptions. Stodden et al. (2016) argue for sharing data, code and workflows so computational results can be verified; the same applies to engineering trials.
For the C-9 trial, the most likely difference between machines is timing: the 40-second health-detection window behind the seed-7 loss. A peer run with a different detection interval could lose more or fewer items, which is why the package must state the tolerance for timing.
47.6 Errors and limits
- Single-run recovery trials.
- Inputs not published or not pinned.
- "Mostly met" reported as met.
- Residual common causes left unstated.
The C-9 trial in the worked example shows why each of these errors matters. A single run with any seed other than 7 would have reported zero loss and hidden the health-detection race. Without published seeds, nobody could rerun seed 7 to confirm the write-behind fix. Reporting zero loss "in most runs" would have shipped the race, and the trial named its common cause: the raw archive shares the storage cluster with the evidence store.
Exercises — chapter 47
EX093 — Package manifest. Objective: define reproducible inputs. Inputs: the worked example. Steps: list every file and parameter in the reproduction package with its purpose and how its integrity is checked. Expected evidence: list. Rubric: 3 = generator, seeds, snapshots with hashes, failure script, environment pins, analysis script, expected outputs; 2 = partial; 1 = vague; 0 = none.
EX094 — How many runs? Objective: reason about rare failures. Inputs: a defect that occurs in a fraction $p$ of runs. Steps: compute the number of runs needed to observe it at least once with 95% probability for $p$ = 0.1 and $p$ = 0.02. Expected evidence: numbers with formula. Rubric: 3 = 29 and 149 runs, with interpretation; 2 = one; 1 = formula; 0 = none.
Chapter synthesis
A recovery trial tests the complete recovery claim with published, pinned inputs, repeated runs and comparison against objectives. Rare failures need many runs; "met in most runs" is a defect or a stated limit, and residual common causes must be named.
Continue studying — chapter 47 of 50
← Chapter 46 · Course chapter index · Chapter 48 →
Continue to the next authored lesson.