Chapter 35 — Design studio: coordinated recovery exercise
Lesson map
Lesson 35 published. Map, example, EX069.
flowchart TD
N0["Learning outcomes and connection"]
N1["35.1 What an exercise must test"]
N2["35.2 Studio brief synthetic"]
N3["35.3 Exercise plan"]
N4["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4
N4 --> EX["Worked example"]
EX --> Q["Exercise EX069"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["35.1 What an exercise must test"]
N2["35.2 Studio brief synthetic"]
N3["35.3 Exercise plan"]
N4["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX069"]
[Learning outcomes and connection]
|
v
[35.1 What an exercise must test]
|
v
[35.2 Studio brief synthetic]
|
v
[35.3 Exercise plan]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX069]
Learning outcomes and connection
After this studio you can (1) design a recovery exercise whose success criteria include service and evidence acceptance, not only uptime; (2) order restoration by dependency and restore point; (3) involve business owners in acceptance; and (4) revise a sequence after a simulated failed restore. It builds on Chapters 13, 26 and 32 and prepares Chapters 40, 43 and 47.
35.1 What an exercise must test
NIST SP 800-84 (2006) describes test, training and exercise programmes — tabletop through functional exercises; its structure remains useful, though tooling examples need updating. NIST SP 800-184 emphasises that recovery plans must be tested and improved. An engineering recovery exercise tests specific claims: the restore sequence works; restore points meet RPO; restored data are clean and complete; dependent services function; the business owner accepts the service as usable; and evidence needed for the investigation survives recovery.
35.2 Studio brief (synthetic)
An HVSS client runs a regional permits portal: web front-end, application API, PostgreSQL database, identity service (federated to the client's IdP) and a message queue for notifications. Scenario: ransomware encrypted the database and application servers; the identity service is suspected but unconfirmed. Objectives: portal read-write within 8 h, at most 1 h of lost permit updates, suspect systems preserved. As in Lab 13, the identity service is suspect and is rebuilt first, from configuration and with new keys. The one-hour bound on lost permit updates is the RPO that the failed restore later had to meet.
35.3 Exercise plan
- Preconditions: isolated recovery environment; backups verified by hash; forensic snapshots of encrypted systems taken before any restore (evidence acceptance).
- Sequence: identity (rebuild from configuration, new keys) → database (restore latest clean backup, replay logs up to the last verified clean point) → queue (fresh; replay undelivered notifications from database outbox) → API → front-end.
- Acceptance criteria per step: identity — test accounts authenticate, no unknown admin roles; database — row counts and checksums match the independent daily export; queue — no duplicate notifications; API — contract tests pass; portal — business owner completes five representative permit transactions.
- Measures: time per step, data loss versus RPO, defects found.
- Fallback per step: if database replay fails, restore the previous backup and record the extra data loss; if identity rebuild fails, use the break-glass local accounts for staff only.
- Roles: exercise director, recovery engineers, business owner, evidence custodian, observer recording timeline.
35.4 Worked example: the failed restore
In the run, the newest database backup (02:00) failed checksum verification — its object had been modified at 02:40, after the intruder obtained backup-service credentials. The fallback restored the previous night's backup and replayed write-ahead logs shipped to a separate account, reaching 03:55, five minutes before encryption began at 04:00. Data loss: within RPO. Time: 9.5 h, exceeding the 8-h objective by 1.5 h, mostly because log replay was untested and slow. Revisions: backups written to immutable storage with separate credentials; log replay rehearsed monthly with timing; the sequence now verifies backup integrity before the identity rebuild finishes, in parallel, so a bad backup is detected earlier.
35.5 Errors and limits
- Success defined as "servers up".
- Restoring before preserving evidence.
- Fallbacks never exercised.
- Business owners absent from acceptance.
The permits-portal run avoided the first, second and fourth errors because the plan required them. Forensic snapshots of the encrypted systems came before any restore, and success meant five permit transactions completed by the business owner. The third error, an unexercised fallback, was the one the run exposed. Write-ahead log replay had never been rehearsed, and that slow step is why the run took 9.5 h instead of 8 h.
Exercises — chapter 35
EX069 — Acceptance criteria. Objective: define acceptance beyond uptime. Inputs: the permits portal. Steps: write measurable acceptance criteria for evidence preservation, data completeness and business usability. Expected evidence: three criteria with checks. Rubric: 3 = specific and checkable for all three; 2 = two; 1 = vague; 0 = none.
EX070 — Parallelise. Objective: shorten the critical path. Inputs: step times: forensic snapshots 1 h, identity rebuild 2 h, backup verification 1 h, database restore + replay 4 h, queue 0.5 h, API 0.5 h, front-end 0.5 h, business acceptance 1 h. Dependencies: snapshots before all restores; database restore after verification; API after database and identity and queue; front-end after API; acceptance last. Steps: compute the critical path with unlimited teams. Expected evidence: path and time. Rubric: 3 = snapshots → verification → database → API → front-end → acceptance = 8.0 h, identity and queue off the critical path; 2 = arithmetic slip; 1 = sequential sum; 0 = none.
Chapter synthesis
A recovery exercise tests claims about sequence, restore points, data cleanliness, service usability and evidence survival. Define acceptance with business owners, rehearse fallbacks and use failed restores to revise the design.
Continue studying — chapter 35 of 50
← Chapter 34 · Course chapter index · Chapter 36 →
Continue to the next authored lesson.