Chapter 47 — Failure and recovery trial
Lesson map
Lesson 47 published. Map, example, EX139.
flowchart TD
N0["Learning outcomes"]
N1["47.1 Script"]
N2["47.2 Measures"]
N3["47.3 Corrective actions"]
N4["47.2b Blameless correction"]
N5["47.3b Recurrence tests"]
N6["47.1b Observer roles"]
N7["Practice note"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX139"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["47.1 Script"]
N2["47.2 Measures"]
N3["47.3 Corrective actions"]
N4["47.2b Blameless correction"]
N5["47.3b Recurrence tests"]
N6["47.1b Observer roles"]
N7["Practice note"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX139"]
[Learning outcomes]
|
v
[47.1 Script]
|
v
[47.2 Measures]
|
v
[47.3 Corrective actions]
|
v
[47.2b Blameless correction]
|
v
[47.3b Recurrence tests]
|
v
[47.1b Observer roles]
|
v
[Practice note]
|
v
[Worked example] --> [Exercise EX139]
Learning outcomes
You will (1) run a scripted failure trial; (2) measure recovery time; (3) file corrective actions.
47.1 Script
Inject: policy file deleted; FindingsDesk killed; clock skew; duplicate event replay. Observe denies, queues, alerts.
The four injects cover policy loss, dependency loss, time and replay — the failure classes that bite engagements. Observing denies, queues and alerts is the scored behaviour. Sam's trial script runs them in a fixed order so measures stay comparable term to term. Skipping an inject to save time fails the trial the same way a missed stop does. Warm-up injects are labelled separately so they never replace scored ones.
47.2 Measures
Time to deny-all after policy loss; time to alert; time to restore; findings lost (should be 0 with queue).
Deny-all time, alert time, restore time and zero lost findings are the SLO inputs. The trial that showed 45s against a 30s target made the poll-interval fix mandatory. Sam records each measure in the same spreadsheet the reliability chapter uses for availability. Spreadsheet rows carry the trial id so terms do not overwrite each other. Overwrites are blocked by making trial id part of the primary key.
47.3 Corrective actions
Each miss becomes a ticket with owner and due date — not only a lesson-learned slide.
"""Verify integrity of a synthetic evidence package."""
import hashlib, json
files = {
"roe.pdf": b"SYNTHETIC-ROE-v1",
"scope.csv": b"host,port\n10.20.1.10,443\n",
"findings.json": b'[{"id":"SYN-001"}]',
}
expected = {n: hashlib.sha256(b).hexdigest()[:16] for n,b in files.items()}
# Simulate tampering of findings.json
current = dict(expected)
current["findings.json"] = hashlib.sha256(b'[{"id":"SYN-001"},{"id":"TAMPERED"}]').hexdigest()[:16]
print(json.dumps(expected, indent=1))
for n in sorted(expected):
status = "OK" if expected[n] == current[n] else "MISMATCH"
print(f"{n:16} {status}")
Expected output:
{
"roe.pdf": "18d74983d403e7db",
"scope.csv": "01cc4815eddf3de6",
"findings.json": "7f5f178ab43ef00f"
}
findings.json MISMATCH
roe.pdf OK
scope.csv OK
Synthetic case — Riverstone Transit. Trial showed 45s to deny-all; target 30s — poll interval tightened.
Lesson-learned slides without tickets do not change poll intervals. Each miss needs an owner and a due date. Sam's 45s result opened a ticket that closed only after two successful recurrence runs at ≤30s. Slide decks may summarise tickets but cannot replace them in the close criteria. Close criteria list ticket ids, not slide titles.
47.4 Errors and limits
Trials on production engagements are forbidden; use synthetic ranges.
Production windows are for authorized assessment, not chaos. Synthetic ranges keep riders out of the blast radius. Sam's trial calendar blocks the synthetic range and refuses a script start if an engagement id is set. A forgotten engagement id in the environment is tested the same way chaos guards are. Script start checks the environment before any inject runs. The check message names the variable so operators know what to unset.
47.2b Blameless correction
Failure trials exist to find system gaps. Write corrective tickets against components and runbooks, not against individuals who followed a bad procedure. Blame-seeking stops people from reporting the next miss.
Ticketing the runbook or the component keeps people willing to report the next miss. Blaming the person who followed a bad step ends learning. Sam's write-up of the 45s miss named the poll interval, not the engineer who started the trial. Names appear only in attendance, never in the root-cause line.
47.3b Recurrence tests
After a corrective fix, re-run the inject that found the gap. Close the ticket only when the measure meets the SLO twice. Otherwise the same failure returns next term.
One lucky green is not an SLO. Two successive meets after the fix are the close rule. Sam's poll-interval ticket stayed open through a flaky first recurrence and closed after the second clean ≤30s run. The flaky first run stayed in the ticket history so patterns remain visible. Patterns across terms feed the next year's trial script review.
47.1b Observer roles
Failure trials include an observer who only notes timings and deviations — not a participant who fixes things live. Fixes come after through tickets.
A participant who fixes live destroys the timing data and hides gaps. Observers only note; fixes go through tickets afterward. Sam's observer sheet for the policy-loss inject shows deny-all at 45s and the alert at 50s without any mid-trial patches. Mid-trial patches void the measures and force a full re-run. Voided runs stay in the archive with a void reason for learning.
Practice note
Failure trials publish a short scoreboard: inject, expected, observed, delta, ticket. Celebrate tickets filed — they are the output. A trial with zero tickets usually skipped hard injects or ignored misses.
Schedule the next trial date before leaving the room so recurrence tests actually happen.
Observers rotate so nobody becomes permanently exempt from being the person under trial.
Scoreboard photos go into the dossier under the failure-trial evidence class.
If an inject cannot be run safely even on synthetic ranges, rewrite it — do not skip and mark green.
47.4b Choosing the failure to rehearse
The trial rehearses the failure with the worst consequence for clients, not the easiest to stage: TrackScope losing its scope file mid-engagement. The script checks that methods stop, the lead is paged and testing resumes only after the file is restored and verified.
Losing the scope file mid-engagement is the worst client consequence on the platform threat list, so it is the rehearsal. The script demands stop, page and verified resume. Sam rejects trial plans that only kill a dashboard pod because they are easier to stage. Easy injects may be used for warm-up but never as the scored rehearsal.
Exercises
- EX139 — Inject. Write one inject and expected safe behaviour. Rubric: 3 = both; 2 = inject only; 1 = vague; 0 = unsafe.
- EX140 — Measure. Primary measure for policy-loss inject. Rubric: 3 = time to deny-all; 2 = vague uptime; 1 = minimal attempt; 0 = none.
- EX141 — Action. Turn a 45s result into a corrective ticket title. Rubric: 3 = specific; 2 = vague; 1 = minimal attempt; 0 = none.
Synthesis
Break it on purpose; measure; ticket the gaps.
Continue studying — chapter 47 of 50
← Chapter 46 (draft) · Course chapter index · Chapter 48 (draft) →