Chapter 47 — Failure and recovery trial
Lesson map
Lesson 47 published. Map, example, EX139.
flowchart TD
N0["Learning outcomes"]
N1["47.1 RTO and RPO"]
N2["Lab 9 — drill scoring"]
N3["47.2 Reading the drills"]
N4["47.3 Improving"]
N5["47.4 Drill safety"]
N6["47.5 Incident linkage"]
N7["Practice note — surprise element"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX139"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["47.1 RTO and RPO"]
N2["Lab 9 — drill scoring"]
N3["47.2 Reading the drills"]
N4["47.3 Improving"]
N5["47.4 Drill safety"]
N6["47.5 Incident linkage"]
N7["Practice note — surprise element"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX139"]
[Learning outcomes]
|
v
[47.1 RTO and RPO]
|
v
[Lab 9 — drill scoring]
|
v
[47.2 Reading the drills]
|
v
[47.3 Improving]
|
v
[47.4 Drill safety]
|
v
[47.5 Incident linkage]
|
v
[Practice note — surprise element]
|
v
[Worked example] --> [Exercise EX139]
Learning outcomes
You will (1) plan recovery-time and recovery-point objectives for Lantern; (2) run failure drills in staging; (3) score drills against objectives; (4) turn misses into improvements.
47.1 RTO and RPO
Recovery time objective (RTO): how long Lantern may be down. Recovery point objective (RPO): how much data may be lost. For BEN, RTO 60 minutes and RPO 15 minutes for the store and ingestion.
The two objectives answer different questions, so they are set separately. An hour of downtime is acceptable because the SOC can fall back on manual checks for a short time. Losing more than fifteen minutes of data would mean indicators that arrived in that window are missing from enforcement until the partner sends them again, which may not happen. The figures were agreed with the SOC lead and written into the requirements.
Lab 9 — drill scoring
"""Lab 9 (ch 47): score a synthetic failure-and-recovery drill against RTO/RPO targets."""
RTO_MIN, RPO_MIN = 60, 15
drills = [("D1 store failover", 42, 5), ("D2 TAXII outage", 75, 0), ("D3 corrupt enrich cache", 30, 25)]
for name, recovery, dataloss in drills:
ok_rto, ok_rpo = recovery <= RTO_MIN, dataloss <= RPO_MIN
print(f"{name:26} RTO {'met' if ok_rto else 'MISSED'} ({recovery}m) RPO {'met' if ok_rpo else 'MISSED'} ({dataloss}m)")
Run with python3 lab_drill.py. Verified output:
D1 store failover RTO met (42m) RPO met (5m)
D2 TAXII outage RTO MISSED (75m) RPO met (0m)
D3 corrupt enrich cache RTO met (30m) RPO MISSED (25m)
47.2 Reading the drills
D1, store failover, met both objectives. D2, a TAXII outage simulation, missed RTO at 75 minutes because the runbook assumed a credential that had rotated. D3, corrupt enrichment cache, recovered quickly but lost 25 minutes of enrichment state, missing RPO.
Kofi presents the misses first in his report, because they are what the drills were for. A drill that passes everything may simply have been too easy. Each miss also had a clear cause that could be fixed, which made the drills worth their cost; the store failover, the case BEN worried about most, worked as designed.
47.3 Improving
D2: update the runbook and add a credential-validity check to the daily health report. D3: snapshot the cache every ten minutes or treat the cache as rebuildable and exclude it from RPO, documenting that choice.
Kofi recorded the choice for the cache as a decision with reasons: rebuilding takes about half an hour from the providers, within their query limits, and no partner data is lost. The SOC lead agreed that enrichment state does not need the same protection as the store. The worked example then confirms that the rerun drill passes under the revised scope.
47.4 Drill safety
Drills run in staging with synthetic data. Partners are not involved unless a joint exercise is agreed, and production enforcement is never altered by a drill.
Kofi's drill plan states the boundaries in writing and the facilitator checks them before starting. The staging environment has its own credentials, which cannot reach production firewalls. A joint exercise with a partner would need a separate agreement covering timing, scope and how to stop if something unexpected happens. Without such care, a drill could cause the outage it was meant to rehearse.
47.5 Incident linkage
NIST SP 800-61 Rev. 3 frames incident response within cybersecurity risk management. Lantern's recovery plans connect to BEN's incident response plan so that an outage during a real incident is handled consistently.
In practice this means Lantern's runbooks name the incident commander as the person who decides priorities if an outage happens during an incident. Kofi's team checked that the two plans use the same contact lists and severity levels. If the store fails while analysts are investigating a campaign, restoring it becomes part of the incident response rather than a separate effort competing for the same people.
Practice note — surprise element
Some drills are announced only to the facilitator, testing the real readiness of on-call engineers rather than rehearsed performance.
Recording drills
Each drill record lists the scenario, start and end times, actions taken, deviations from the runbook and follow-up actions with owners. Records are kept for audits and for comparing readiness across quarters.
Worked example
After fixes, D2 is rerun: recovery in 38 minutes. D3's cache is reclassified as rebuildable; the RPO now applies only to the store, which met it.
Synthetic case. Drill results are invented.
Errors and limits
Redefining objectives to pass drills must be a documented decision, not a convenience. Staging drills miss some production behaviours.
Exercises
- EX139 — Drill misses. Which drill missed RTO and which missed RPO? Rubric: 3 = D2 RTO; D3 RPO; 2 = one; 1 = wrong; 0 = none.
- EX140 — RTO vs RPO. Define RTO and RPO in one line each. Rubric: 3 = max downtime; max data loss; 2 = one; 1 = swapped; 0 = none.
- EX141 — Reclassify honestly. When is reclassifying the cache as rebuildable legitimate? Rubric: 3 = documented decision with justification that loss is harmless/rebuildable; 2 = documented; 1 = to pass drill; 0 = none.
Synthesis
Set RTO and RPO, drill safely, score honestly, fix causes and document any change to objectives.
Continue studying — chapter 47 of 50
← Chapter 46 (draft) · Course chapter index · Chapter 48 (draft) →