Chapter 47 — Failure and recovery trial
Lesson map
Lesson 47 published. Map, example, EX139.
flowchart TD
N0["Learning outcomes"]
N1["47.1 Trial plan"]
N2["47.2 Safety"]
N3["47.3 Leaked credential scenario"]
N4["47.4 Deleted policy scenario"]
N5["47.5 Engine failure scenario"]
N6["47.6 Results"]
N7["Debrief"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX139"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["47.1 Trial plan"]
N2["47.2 Safety"]
N3["47.3 Leaked credential scenario"]
N4["47.4 Deleted policy scenario"]
N5["47.5 Engine failure scenario"]
N6["47.6 Results"]
N7["Debrief"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX139"]
[Learning outcomes]
|
v
[47.1 Trial plan]
|
v
[47.2 Safety]
|
v
[47.3 Leaked credential scenario]
|
v
[47.4 Deleted policy scenario]
|
v
[47.5 Engine failure scenario]
|
v
[47.6 Results]
|
v
[Debrief]
|
v
[Worked example] --> [Exercise EX139]
Learning outcomes
You will (1) plan and run a controlled failure and recovery trial; (2) test controls and recovery under realistic stress; (3) measure recovery; (4) learn from results.
47.1 Trial plan
With approval from Solent's operations director, Elena and the platform team run a staged trial in a production-like environment: simulate a leaked workload credential, a deleted network policy and an unavailable policy engine, then measure detection and recovery.
Each scenario tests a different layer: detection and revocation for identity, reconciliation for configuration, and fail-closed behaviour for admission. Objectives are written in advance as times, such as detection within five minutes, so results can be judged against them rather than described afterwards. The production-like environment uses the same templates, policies and alerting as production, which makes the results meaningful without putting real meter data or customers at risk.
47.2 Safety
The trial uses a dedicated environment with synthetic meter data. Abort criteria and a rollback plan are agreed; observers are named.
Abort criteria are specific: stop if any production alert fires, if the trial environment's isolation is in doubt, or if an observer calls a halt. The rollback plan restores the environment from its declared state in the repository. Observers record timings and decisions without taking part, which keeps measurements independent of the people being measured. Production on-call staff are told the trial window in advance, so trial alerts reaching them are not mistaken for real incidents.
47.3 Leaked credential scenario
A synthetic credential event is injected. Detection takes 4 minutes; the credential is revoked in 9 minutes. Because workload identity tokens expire within an hour, the scenario also confirms that no long-lived key exists to revoke.
Four minutes to detect and nine to revoke are both inside the objectives. The leaked credential is a short-lived workload token, so even without revocation it would stop working within the hour; revoking it still matters because an hour is long enough to copy data. The confirmation that no long-lived key exists is the strongest result, since the opening case began with exactly such a key.
47.4 Deleted policy scenario
A network policy is deleted. GitOps restores it in 3 minutes; a drift alert fires in 2.
The alert arriving before the restore matters: if GitOps reverted the change silently, nobody would ask why the policy was deleted, and a person or process deleting policies would go unnoticed. During the three minutes the namespace was open, observers check flow logs for connections that the policy would have blocked; none appear in the synthetic traffic. The trial also confirms that reconciliation works when the deletion happens outside the GitOps repository.
47.5 Engine failure scenario
All policy engine replicas are stopped. Deployments fail closed; an alert fires in 1 minute; recovery takes 6 minutes following the runbook.
Failing closed means no new deployment is admitted while the engine is down; running workloads continue unaffected. The cost is blocked releases, which is why recovery speed matters. Six minutes meets the objective, and following the runbook step by step shows where it was unclear. The worked example shows the human side of the scenario: under pressure, an engineer considered making admission fail open, which the runbook now forbids without approval.
47.6 Results
All objectives met; two runbook steps clarified.
Detection and recovery times were inside every objective: four and nine minutes for the credential, two and three for the deleted policy, and one and six for the engine failure. The two clarified runbook steps were the points where engineers paused during the engine failure. Elena records the results with the trial plan and the observers' notes, so next year's trial can compare against them, and she notes that synthetic data cannot show everything a real incident would.
Debrief
The trial ends with a debrief within 24 hours: what went well, what surprised the team and what changes follow. Findings are tracked like incidents, with owners and dates. The two runbook clarifications are made the same week, and the fail-open rule is added to on-call training. The team agrees to run a similar trial each year with new scenarios chosen from the threat register.
Approval
The trial plan was approved in writing a week ahead.
Worked example
An observer notes that during the engine failure, an engineer considered switching admission to fail open to unblock a deployment. The runbook now states explicitly that fail-open in production requires the platform lead's approval and an incident record.
Synthetic case. Trial and timings are synthetic.
Errors and limits
Trials are only as realistic as their scenarios; vary them each year.
Exercises
- EX139 — Safety measures. How was the trial kept safe? Rubric: 3 = dedicated environment, synthetic data, abort criteria, observers; 2 = two; 1 = one; 0 = none.
- EX140 — Credential times. What were detection and revocation times? Rubric: 3 = 4 and 9 minutes; 2 = partial; 1 = vague; 0 = none.
- EX141 — Runbook change. What runbook change followed? Rubric: 3 = fail-open needs approval and incident; 2 = partial; 1 = vague; 0 = none.
Synthesis
Controlled trials prove recovery works under stress.
Continue studying — chapter 47 of 50
← Chapter 46 (draft) · Course chapter index · Chapter 48 (draft) →