Chapter 47 — Failure and recovery trial
Lesson map
Lesson 47 published. Map, example, EX139.
flowchart TD
N0["Learning outcomes"]
N1["47.1 The trial"]
N2["47.2 Expectations"]
N3["47.3 Observations"]
N4["47.4 Recovery"]
N5["47.5 Lessons"]
N6["47.6 Measuring"]
N7["Safety of the trial"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX139"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["47.1 The trial"]
N2["47.2 Expectations"]
N3["47.3 Observations"]
N4["47.4 Recovery"]
N5["47.5 Lessons"]
N6["47.6 Measuring"]
N7["Safety of the trial"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX139"]
[Learning outcomes]
|
v
[47.1 The trial]
|
v
[47.2 Expectations]
|
v
[47.3 Observations]
|
v
[47.4 Recovery]
|
v
[47.5 Lessons]
|
v
[47.6 Measuring]
|
v
[Safety of the trial]
|
v
[Worked example] --> [Exercise EX139]
Learning outcomes
You will (1) plan a controlled failure trial of the pipeline and verifier; (2) observe detection and recovery; (3) measure recovery time; (4) record and act on lessons.
47.1 The trial
With approval, Coastline runs a planned trial in staging: a synthetic tampered artifact is placed on the mirror, and the signing service is made unavailable for two hours.
The two failures test different properties. The tampered artifact tests integrity controls: would anything bad reach a server? The signing outage tests failing closed and recovery: do builds stop safely, and do they resume cleanly? Approval names the trial lead, the time window and the systems involved, and only staging is touched.
47.2 Expectations
The deployment should refuse the tampered artifact; the verifier should alert; builds should wait during the signing outage and resume afterwards.
Writing expectations before the trial turns it into a test rather than a demonstration. Each expectation can be observed and timed, so the result is clearly pass or fail. If something unexpected happens, the gap between expectation and observation is the lesson. Expectations come from the requirements and the control design, so the trial also checks that the design was implemented as intended.
47.3 Observations
The tampered artifact is refused within seconds and the alert reaches on-call in 3 minutes. During the signing outage, builds queue as designed, but one team, unaware of the trial, tries to bypass signing with a manual flag; the policy blocks it.
The alert time comes from the moment the artifact was placed, not when someone noticed. Refusal in seconds shows the digest check ran before any installation step. The blocked manual flag is the surprise: the trial was not meant to test it, yet it showed that a bypass still existed in the configuration and that the policy caught its use. Observations are logged with timestamps by an observer who has no other role.
47.4 Recovery
After the outage, queued builds complete within 25 minutes. The mirror is cleaned and its digests rechecked.
Twenty-five minutes reflects the backlog of builds that piled up during two hours, processed with normal capacity. Rechecking every digest on the mirror confirms that only the synthetic artifact was affected and that cleaning it left nothing behind. Recovery counts as finished only when the system is back to a verified state, not when builds simply restart. The team also checks that no build skipped signing during the outage by comparing the build log with the transparency log.
47.5 Lessons
Communication: teams need notice of trials and a status page during outages. The manual flag is removed entirely.
The team that tried the manual flag was acting reasonably with the information it had: builds were stuck and nobody had said why. Telling all teams in advance and showing outages on a status page would have prevented the attempt. Removing the flag altogether means no one can try it in future, whether during a trial or a real outage. The emergency path described in earlier chapters, with named approval, remains the only route for urgent fixes.
47.6 Measuring
Time to detect: 3 minutes; time to recover: 25 minutes after service return.
These two numbers give the yearly trial a baseline. If detection takes longer next year, the team will look at alert routing; if recovery takes longer, at build capacity or queue handling. Measuring recovery from service return separates Coastline's response from the length of the outage, which Coastline did not control. Both figures are added to the evidence dossier, with the trial date beside them, so next year's result can be compared directly.
Safety of the trial
The trial ran only in staging, with production untouched. A named trial lead could stop it at any time, and all affected teams except one were told in advance; that missed notice became the main lesson. The tampered artifact was synthetic and clearly labelled, and it was deleted and its removal verified after the trial.
Follow-up
Actions from the trial are tracked to closure, with owners and dates, and reviewed one month later.
Repeatability
The written procedure lists preparation, roles, steps, stop conditions and clean-up, so the platform team can repeat the trial each year.
Worked example
Ingrid adds the trial to the yearly calendar and asks Kwame to write the procedure so others can repeat it after he leaves.
Synthetic case. The trial is synthetic.
Errors and limits
Staging trials may not reproduce production load or human behaviour exactly.
Exercises
- EX139 — Trial setup. What two failures were introduced? Rubric: 3 = tampered artifact on mirror; signing service down 2 h; 2 = one; 1 = vague; 0 = none.
- EX140 — Bypass attempt. What happened when a team tried to bypass signing? Rubric: 3 = policy blocked it; flag removed; 2 = partial; 1 = vague; 0 = none.
- EX141 — Measures. Give detection and recovery times. Rubric: 3 = 3 min; 25 min; 2 = one; 1 = vague; 0 = none.
Synthesis
Planned failure trials prove that controls hold and recovery works.
Continue studying — chapter 47 of 50
← Chapter 46 (draft) · Course chapter index · Chapter 48 (draft) →