Chapter 40 — Verification lab: failure injection and recovery
Lesson map
Lesson 40 published. Map, example, EX079.
flowchart TD
N0["Learning outcomes and connection"]
N1["40.1 Fault injection as verification"]
N2["40.2 Plan for this lab"]
N3["40.3 The lab"]
N4["40.4 Comparing with claims"]
N5["40.5 Reproducibility and reporting"]
N6["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX079"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["40.1 Fault injection as verification"]
N2["40.2 Plan for this lab"]
N3["40.3 The lab"]
N4["40.4 Comparing with claims"]
N5["40.5 Reproducibility and reporting"]
N6["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX079"]
[Learning outcomes and connection]
|
v
[40.1 Fault injection as verification]
|
v
[40.2 Plan for this lab]
|
v
[40.3 The lab]
|
v
[40.4 Comparing with claims]
|
v
[40.5 Reproducibility and reporting]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX079]
Learning outcomes and connection
After this lab you can (1) plan a bounded fault-injection experiment against an isolated response service; (2) state hypotheses, stop conditions and observations before running; (3) compare observed behaviour with design claims; and (4) make the experiment reproducible. It builds on Chapters 28, 30, 32 and 33 and prepares Chapters 46 and 47.
40.1 Fault injection as verification
Fault injection deliberately introduces failures to test error handling. Chaos engineering applies the idea to running distributed systems with explicit hypotheses about steady-state behaviour (Basiri et al., 2016). For response systems the rule is strict: never in production evidence or control paths without explicit authorisation and safeguards — use an isolated environment with synthetic data, or a production-like staging copy. The experiment has a written plan: hypothesis, fault, scope, stop conditions, observations, rollback and reproducibility details. Lab 40 below follows that rule by simulating the outage entirely in code.
40.2 Plan for this lab
- System: an isolated workflow service writing evidence records to an evidence store.
- Claim under test: design O (local outbox with retry and idempotent keys) loses no records during an evidence-store outage of up to 15 minutes, while design N (direct writes) does.
- Fault: evidence store unavailable for minutes 20–35.
- Stop condition: outbox exceeds 25 records — the experiment is aborted and the store restored, to bound resource use and to show how a long outage is handled.
- Observations: records stored, lost, left in outbox; stop events.
- Reproducibility: deterministic simulation; parameters in code.
40.3 The lab
"""Lab 40 (synthetic): bounded fault injection. Evidence store unavailable for minutes 20-35.
Design N writes directly; design O uses a local outbox with retry, idempotent keys and a stop condition."""
def run(design, outage=(20, 35), n=60, stop_outbox=25):
store, outbox, lost, dup, stopped, log = {}, [], 0, 0, None, []
up = lambda t: not (outage[0] <= t < outage[1])
for t in range(n):
rec = (f"ev{t}", f"payload{t}")
if design == "N":
if up(t): store[rec[0]] = rec[1]
else: lost += 1
else:
outbox.append(rec)
if len(outbox) > stop_outbox and stopped is None:
stopped = t; log.append(f"t={t} STOP CONDITION: outbox > {stop_outbox}; experiment aborted, store restored")
outage = (outage[0], t)
if up(t):
for k, v in outbox:
if k in store: dup += 1
store[k] = v
outbox = []
if t == outage[1]: log.append(f"t={t} store back; outbox flushed")
return len(store), lost, len(outbox), stopped, log
for d in ("N", "O"):
s, lost, ob, st, log = run(d)
print(f"design {d}: stored {s}/60 lost {lost} left in outbox {ob} stop={st}")
for l in log: print(" ", l)
s, lost, ob, st, log = run("O", outage=(20, 70))
print(f"design O, long outage: stored {s}/60 lost {lost} left {ob} stop={st}")
for l in log: print(" ", l)
Expected output:
design N: stored 45/60 lost 15 left in outbox 0 stop=None
design O: stored 60/60 lost 0 left in outbox 0 stop=None
t=35 store back; outbox flushed
design O, long outage: stored 60/60 lost 0 left 0 stop=45
t=45 STOP CONDITION: outbox > 25; experiment aborted, store restored
t=45 store back; outbox flushed
The lab runs a 60-minute simulation, producing one evidence record per minute and making the evidence store unavailable from minute 20 to minute 35. Design N writes each record straight to the store and loses whatever arrives while the store is down. Design O appends to a local outbox, flushes it whenever the store is reachable, and counts any key it writes twice. A third run stretches the outage to minute 70 so that the outbox crosses its limit of 25 records and the stop condition fires. Because the simulation uses no randomness, the expected output below is reproduced exactly on every run.
40.4 Comparing with claims
Design N loses exactly the 15 records produced during the outage: the claim that N is unsafe is confirmed. Design O stores all 60 records with no loss and flushes its outbox when the store returns at minute 35: the claim holds for a 15-minute outage. In the long-outage variant, the outbox reaches the stop threshold at minute 45; the stop condition fires, the store is restored and all 60 records are eventually stored. This demonstrates the stop condition but also exposes an untested assumption: in a real long outage nobody can "restore the store" on demand. The design therefore needs a second-tier behaviour — spill the outbox to durable local disk and alert operators — and a new experiment to verify it. Fault injection's most valuable output is often a claim you discover you had not tested.
40.5 Reproducibility and reporting
Report: environment, versions, parameters, seed (if random), timeline of fault and observations, results against each hypothesis, deviations from plan, and follow-up actions. Store the experiment definition in version control so the same test can be rerun after changes (regression) — the basis of the trial in Chapter 47. Sandve et al. (2013) give simple rules for reproducible computational work that transfer directly.
For Lab 40, the report would record the outage window of minutes 20–35 and the stop limit of 25 records. It would show the 45/60 and 60/60 results for designs N and O, and list the deviation that matters. The long-outage stop condition at minute 45 assumed the store could be restored on demand, and spilling to durable disk becomes the follow-up experiment.
40.6 Errors and limits
- Injecting faults in production without authorisation and stop conditions.
- No hypothesis written before the run.
- Stop conditions that assume capabilities unavailable in real failures.
- Simulations taken as proof of real-system behaviour without staging tests.
Lab 40 itself shows the third and fourth errors in this list. Its stop condition "restores the store" at minute 45, something no operator can do during a real long outage, and the run looks clean only because of that. The 60/60 result for design O comes from a deterministic simulation. It must be repeated against the staging copy of the workflow service before anyone states the claim for the real system.
Exercises — chapter 40
EX079 — Duplicate safety. Objective: extend the experiment. Inputs: Lab 40. Steps: modify design O so that the store sometimes acknowledges a write but the acknowledgement is lost (the record is stored but stays in the outbox and is resent); show that idempotent keys prevent duplicates and report the dup counter. Expected evidence: code and output. Rubric: 3 = correct injection, stored count still 60, duplicates detected as overwrites of same key, explanation; 2 = output only; 1 = attempt; 0 = none.
EX080 — Experiment plan. Objective: plan a real staging test. Inputs: HVSS staging environment with the workflow service, evidence store and pager. Steps: write a one-page plan for a 30-minute evidence-store outage test: hypothesis, scope, authorisation, stop conditions, observations, rollback, communication. Expected evidence: plan. Rubric: 3 = complete with real stop conditions and authorisation; 2 = one gap; 1 = vague; 0 = none.
Chapter synthesis
Fault injection verifies resilience claims when it is bounded, hypothesis-driven, isolated from production evidence and control, stopped by pre-agreed conditions and reproducible. Its findings include the claims you had not yet tested.
Continue studying — chapter 40 of 50
← Chapter 39 · Course chapter index · Chapter 41 →
Continue to the next authored lesson.