OpenCyber Academy
Postgraduate · Chapter 6 · English
Course code: OCA-154Chapter 06: OCA-154-C006

Chapter 6 — Controlled evaluation of recovery strategies

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 06 published. Map, example, EX026.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["6.1 Recovery as an experiment"]
  N2["6.2 Validation references"]
  N3["6.3 Downstream synchronization"]
  N4["6.4 Acceptance and fallback"]
  N5["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX026"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["6.1 Recovery as an experiment"]
  N2["6.2 Validation references"]
  N3["6.3 Downstream synchronization"]
  N4["6.4 Acceptance and fallback"]
  N5["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX026"]

[Learning outcomes and connection]
    |
    v
[6.1 Recovery as an experiment]
    |
    v
[6.2 Validation references]
    |
    v
[6.3 Downstream synchronization]
    |
    v
[6.4 Acceptance and fallback]
    |
    v
[Chapter synthesis]
    |
    v
[Worked example] --> [Exercise EX026]

Learning outcomes and connection

You will be able to (1) design a controlled comparison of recovery strategies before committing production; (2) validate restored data against independent references; (3) handle uncertainty in downstream synchronization; (4) define acceptance and fallback criteria; and (5) report the evaluation's limitations. Chapter 5 contained the adversary; this chapter returns services safely. Chapter 8 deals with conflicting evidence, which often surfaces during recovery validation.

6.1 Recovery as an experiment

Specialists rarely have one obvious recovery path. Restoring a database from a verified backup may lose recent transactions; replaying logs may reintroduce attacker changes; rebuilding from upstream systems may be incomplete. Rather than choose by intuition, run a controlled evaluation: restore candidates into isolated environments, measure them against the same validation suite, and compare. NIST SP 800-184 emphasizes planning, testing and improvement of recovery; SP 800-209 discusses storage infrastructure security including backup protection.

Calder Logistics faced exactly this when its order database was encrypted: a verified nightly backup or a more recent but unverified replica.

6.2 Validation references

Validate restored data against references that the adversary did not control: upstream systems of record, partner records, financial reconciliations, previously exported reports with hashes, and samples confirmed by business users. Each reference has its own errors; disagreements must be investigated, not averaged.

At Calder, the warehouse management system was the reference: it independently recorded 1,000 order events between 02:00 and 10:03, including 40 orders flagged as created by the adversary. Counting each candidate against that set showed what each one contained, which neither candidate could show about itself.

6.3 Downstream synchronization

Restored systems feed others: a restored order database must re-synchronize with warehouse, invoicing and customer-notification systems. If downstream systems processed data after the restore point, the two can diverge (duplicate shipments, missing invoices). Map the synchronization paths and decide, per path, whether to replay, reconcile or freeze.

Calder's order database feeds warehouse dispatch and invoicing. Restoring the 02:00 backup and reconstructing 960 legitimate orders from the warehouse system meant invoicing had to be reconciled against the rebuilt set, which is why a zero invoicing difference is one of the acceptance criteria.

6.4 Acceptance and fallback

Acceptance criteria: data validation thresholds, functional tests, security checks, performance, and owner sign-off. Fallback: if a criterion fails after go-live, what is the path — revert to manual operation, switch to the alternative candidate, or partial operation?

Calder's criteria for the order database were specific: the restored base hash matches the backup manifest; reconstructed orders equal the warehouse system's 960 legitimate orders minus customer cancellations; zero orders go to the fraud drop address; the invoicing difference is zero; and the owner signs off. The fallback was manual dispatch from the warehouse system for 24 hours if reconstruction failed validation.

6.5 Worked example and lab (synthetic): two restore candidates

Fictional. Calder Logistics' order database was encrypted. Candidate R1: nightly backup from 02:00 (verified hash, immutable storage). Candidate R2: an asynchronous replica whose replication stopped at 09:41, 22 minutes before encryption at 10:03 — more recent but not verified; the adversary had access to the primary from 08:15.

Validation reference: the warehouse management system (WMS) independently recorded 1,000 order events between 02:00 and 10:03, including 40 orders the investigation flagged as created by the adversary (fraudulent orders to a drop address). Save as lab06.py.

# Lab PG-06 - compare two restore candidates against an independent reference (synthetic, stdlib only)
import random
random.seed(6)

# Reference (WMS) orders 02:00-10:03: ids 1..1000; ids 601..640 fraudulent (created 08:20-09:30)
legit = set(range(1, 601)) | set(range(641, 1001))
fraud = set(range(601, 641))
order_minute = {i: 120 + (i * 483) // 1000 for i in range(1, 1001)}   # minutes after midnight, 02:00..10:03
replica_stop = 9*60 + 41
R1 = set()                                                           # nightly backup: none of today's orders
R2 = {i for i in range(1, 1001) if order_minute[i] <= replica_stop}
R2 -= set(random.sample(sorted(R2), 7))                              # replication lag losses
for name, cand in (("R1 nightly", R1), ("R2 replica", R2)):
    have_legit = len(cand & legit)
    have_fraud = len(cand & fraud)
    to_reenter = len(legit - cand)
    print(f"{name}: legit present {have_legit}, fraudulent present {have_fraud}, legit to re-enter {to_reenter}")

Expected output:

R1 nightly: legit present 0, fraudulent present 0, legit to re-enter 960
R2 replica: legit present 909, fraudulent present 40, legit to re-enter 51

Interpretation: R2 saves re-entry of about 900 legitimate orders but contains all 40 fraudulent orders and lost 7 records in replication; it also carries any undiscovered attacker changes made between 08:15 and 09:41. R1 is clean with respect to today's activity but requires re-entering 960 orders from the WMS. A defensible hybrid is: restore R1 (trusted base), then reconstruct today's orders from the WMS (the independent reference), excluding the 40 flagged fraudulent orders and holding for review any order whose attributes match fraud patterns. This uses R2 only as a cross-check, not as a source. The lab's numbers depend on the synthetic generator; the method — counting what each candidate contains against an independent reference — is the point.

Acceptance criteria (excerpt): restored base hash matches backup manifest; reconstructed orders equal WMS legit set (960) minus orders cancelled by customers; zero orders to the fraud drop address; invoicing reconciliation difference zero; owner sign-off. Fallback: if WMS reconstruction fails validation, continue manual dispatch from WMS for 24 hours while investigating.

6.6 Common errors and limits

Calder's replica R2 shows the first error. It was 22 minutes more recent than encryption at 10:03, but it contained all 40 fraudulent orders and any undiscovered attacker changes made after 08:15. Choosing it by default would have restored the attacker's work. Validating against the warehouse system, not against the database itself, is what exposed the difference, and even that reference needs its flagged orders reviewed, not trusted blindly.

Exercises — chapter 6

Attempt each exercise before opening the worked results. Every case, organization, person, address and number in an exercise is synthetic unless a cited public source is named.

EX026 — Run and explain. Steps: run Lab PG-06; change the seed to 7 and report what changes and what does not. Rubric: 3 = output reported; explains that only which 7 replication losses occur changes, counts stay (909/40/51) because the sample size is fixed; 2 = output only; 1 = not run; 0 = none.

EX027 — Synchronization map. Steps: map downstream paths from the order database (warehouse, invoicing, customer notifications, analytics) and choose replay, reconcile or freeze for each. Rubric: 3 = each path with choice and reason (e.g., freeze notifications until reconciled to avoid messaging fraudulent orders); 2 = choices without reasons; 1 = list only; 0 = none.

EX028 — Acceptance criteria. Steps: complete the acceptance criteria and fallback for the hybrid. Rubric: 3 = data, function, security, performance, sign-off; fallback with trigger; 2 = missing security or trigger; 1 = vague; 0 = none.

EX029 — Evaluation report. Steps: write a 150-word report of the comparison for the commander, including limitations. Rubric: 3 = method, results, recommendation, limitations (synthetic generator, reference reliability, undiscovered changes); 2 = no limitations; 1 = opinion; 0 = none.

EX030 — When to choose R2. Steps: describe conditions under which R2 would be the better base. Rubric: 3 = attacker access start verified after replica stop, or strong integrity verification of replica, and re-entry infeasible; 2 = one condition; 1 = vague; 0 = none.

Chapter synthesis

Treat recovery choices as controlled comparisons: restore candidates in isolation, validate against independent references, map downstream synchronization, define acceptance and fallback — and prefer a clean base rebuilt from trusted references over a recent but untrusted copy.

Worked answers

EX026. With seed 7 the printed counts are unchanged: R1 0/0/960; R2 909/40/51. The seed only chooses which seven replicated orders are lost; the number lost is fixed, and in both seeds none of the seven is a fraudulent order (a different seed could select one, giving 39 fraudulent present). The recommendation must not depend on such generator detail: what matters is that the replica contains the fraudulent orders and possibly undiscovered changes, while the nightly backup does not.

EX027. Warehouse (WMS): reconcile — WMS is the reference; restored orders must match it. Invoicing: freeze until reconciliation, then replay from the reconstructed order set, to avoid invoices for fraudulent orders. Customer notifications: freeze; resume only for reconstructed legitimate orders to avoid confusing or alerting the fraud drop address. Analytics: rebuild after recovery from the final data; mark the incident window.

EX028. Data: base hash matches manifest; reconstructed legitimate orders equal WMS legit set; zero orders to fraud drop address or matching fraud patterns without review. Function: create, update, cancel and ship a test order in isolation. Security: database credentials rotated; no adversary accounts; monitoring for the drop address. Performance: response times within 20% of baseline under test load. Sign-off: Logistics service owner. Fallback: if reconciliation differs by more than 0.5% or invoicing fails, revert to manual WMS dispatch for 24 hours.

EX029. "We compared two restore candidates against the warehouse system's independent record of 1,000 orders made today. The nightly backup is clean but requires re-entering 960 legitimate orders; the replica is more recent but contains all 40 fraudulent orders, lost 7 records, and may hold undiscovered attacker changes since 08:15. We recommend restoring the nightly backup and rebuilding today's orders from the warehouse record, excluding flagged fraud and reviewing suspicious matches. Limits: the warehouse record may contain errors; fraud flags may be incomplete; the comparison uses one reference."

EX030. R2 would be preferable if (a) the investigation verified that adversary access to the primary began after the replica stopped, or (b) the replica's contents can be verified record-by-record against trusted references and any adversary changes removed with high confidence, and (c) re-entering orders from other sources is infeasible within the recovery-time objective.