Chapter 47 — Failure and recovery trial
Lesson map
Lesson 47 published. Map, example, EX139.
flowchart TD
N0["Learning outcomes"]
N1["47.1 Script"]
N2["47.2 Measures"]
N3["47.3 Corrective actions"]
N4["Practice note"]
N5["47.1b Blameless correction and recurrence"]
N6["Field note — zero-ticket smell"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX139"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["47.1 Script"]
N2["47.2 Measures"]
N3["47.3 Corrective actions"]
N4["Practice note"]
N5["47.1b Blameless correction and recurrence"]
N6["Field note — zero-ticket smell"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX139"]
[Learning outcomes]
|
v
[47.1 Script]
|
v
[47.2 Measures]
|
v
[47.3 Corrective actions]
|
v
[Practice note]
|
v
[47.1b Blameless correction and recurrence]
|
v
[Field note — zero-ticket smell]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX139]
Learning outcomes
You will (1) run a scripted failure trial; (2) measure recovery time; (3) file corrective actions.
47.1 Script
Inject: allow-list file deleted; finding store killed; clock skew; duplicate event replay; flat graph substituted for zoned. Observe denies, queues, alerts, reachability fail.
In ABP's bench failure trial, Lena's team deleted the allow-list file while replayed traffic was running, then stopped the finding store, skewed the clock on the bench copy of PLC-A and replayed duplicate events. Finally they substituted a flat graph for the zoned one. Observers recorded whether writes were denied, whether findings queued, whether alerts fired and whether the reachability check failed on the flat graph. Each inject was run separately before any combination was attempted.
47.2 Measures
Time to deny-all writes after policy loss; time to alert; time to restore; findings lost (should be 0 with queue); SIS reachability check still fails closed on bad graph.
At ABP the trial measured 45 seconds to deny all writes after the allow-list was deleted, against a 30-second target. Alerts fired within the target, and no findings were lost, because drafts queued while the finding store was down. The reachability check correctly failed when the flat graph was substituted, showing SIS-B as reachable. Each measure was recorded by an observer with a stopwatch and checked against the logs, so the numbers did not rely on anyone's memory.
47.3 Corrective actions
Each miss in a CNHP drill becomes a ticket with owner and due date — not only a lesson-learned slide.
# Lab 27 - residual CUSUM vs band on batch reactor level (synthetic)
import random
def run(seed, atk, det, k=0.8, h=6.0):
rng=random.Random(seed); true=est=50.0; s=0.0; alarm=None
for t in range(500):
inflow=1.0+rng.gauss(0,0.05)+(0.08 if atk and t>=250 else 0)
true+=inflow-0.02*true
lvl=true+rng.gauss(0,0.3); fm=inflow+rng.gauss(0,0.02)
if atk and t>=250: lvl=50+rng.gauss(0,0.3)
if atk=="B" and t>=250: fm=1.0+rng.gauss(0,0.02)
est+=fm-0.02*est
hit=abs(lvl-50)>2.0 if det=="band" else False
if det=="cusum":
s=max(0.0,s+abs(lvl-est)-k); hit=s>h
if hit and alarm is None: alarm=t
return alarm
for det in ("band","cusum"):
fa=sum(run(i,None,det) is not None for i in range(60))
for atk in ("A","B"):
hits=[a-250 for a in (run(i,atk,det) for i in range(60)) if a is not None and a>=250]
print(f"{det} class{atk} FA={fa}/60 det={len(hits)}/60 mean_delay={sum(hits)/len(hits) if hits else float('nan'):.1f}")
Expected output:
band classA FA=0/60 det=0/60 mean_delay=nan
band classB FA=0/60 det=0/60 mean_delay=nan
cusum classA FA=0/60 det=60/60 mean_delay=24.8
cusum classB FA=0/60 det=0/60 mean_delay=nan
Synthetic case — Ashford Batch Plant. Trial showed 45s to deny-all writes; target 30s — poll interval tightened.
At ABP the 45-second deny time became a ticket for the monitor's owner with a due date. The fix shortened the policy check interval, and the inject was rerun until it met the target twice. The ticket closed only after the second passing run, and the next trial date was booked before the observers left the room.
47.4 Errors and limits
Trials on production windows are forbidden; use synthetic ranges.
Lena's trials at ABP run on the bench range with synthetic traffic, never during a production window. Deleting the allow-list on the live monitor would test the same code path but could interrupt engineering writes while a batch was running on R-101. The bench copy of PLC-A and the replayed trunk traffic give the timing evidence needed. Production checks after a fix are read-only, such as confirming that the deployed version matches the one tested on the bench.
Practice note
Celebrate tickets filed — they are the output. A trial with zero tickets usually skipped hard injects.
47.1b Blameless correction and recurrence
Corrective tickets name the monitor, conduit or runbook at fault, not the operator who followed a bad procedure. After a fix, re-run the inject that found the gap. Close the ticket only when the measure meets the SLO twice. Observers rotate so nobody is permanently exempt. A ticket that names a person instead of a component goes back to its author for rewording before it is accepted.
Observers note timings and deviations — they do not fix things live. Their timings are checked against the logs before the scoreboard is photographed. Scoreboard photos go into the dossier.
Field note — zero-ticket smell
A trial that produced zero tickets had skipped the flat-graph inject. Re-running with the hard inject filed three tickets and tightened the SIS reachability gate. The flat-graph inject is now mandatory in every trial.
Each corrective ticket links the inject script and the measured value that triggered it.
Rotated observers are briefed on the measures before the trial, not during it.
47.5 Worked example — recovery trial on the bench (synthetic ABP)
The trial scenario: PLC-A's logic is corrupted (simulated by loading a wrong program on the bench controller) during a batch. Measures: time to detect (program-change alert 35 s), time to decide (shift lead chooses safe hold, 4 min), time to restore verified logic from the offline backup (22 min) and time to resume (41 min total). The trial reveals that the backup's hash had never been recorded, so verification relied on a manual comparison. Corrective actions get owners and dates: record backup hashes, add a restore drill to the quarterly calendar, and track recurrence.
Exercises
- EX139 — Inject. One inject and expected safe behaviour. Rubric: 3 = both; 2 = inject only; 1 = vague; 0 = unsafe.
- EX140 — Measure. Primary measure for policy-loss inject. Rubric: 3 = time to deny writes; 2 = vague uptime; 1 = minimal attempt; 0 = none.
- EX141 — Action. Turn a 45s result into a corrective ticket title. Rubric: 3 = specific; 2 = vague; 1 = minimal attempt; 0 = none.
Synthesis
Break it on purpose; measure; ticket the gaps.
Continue studying — chapter 47 of 50
← Chapter 46 (draft) · Course chapter index · Chapter 48 (draft) →