Chapter 11 — Supervised incident simulation

Lesson map

Lesson 11 published. Map, example, EX051.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["11.1 Design a test of decisions, not a dramatic stor"]
  N2["11.2 Facilitator packet and release ledger"]
  N3["11.3 A decision log that a second shift can reconstr"]
  N4["11.4 Score the artifact, then discuss the decision"]
  N5["11.5 Run a negative control separately"]
  N6["Practice: supervised 70-minute tabletop"]
  N7["Subject synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX051"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["11.1 Design a test of decisions, not a dramatic stor"]
  N2["11.2 Facilitator packet and release ledger"]
  N3["11.3 A decision log that a second shift can reconstr"]
  N4["11.4 Score the artifact, then discuss the decision"]
  N5["11.5 Run a negative control separately"]
  N6["Practice: supervised 70-minute tabletop"]
  N7["Subject synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX051"]

[Learning outcomes and connection]
    |
    v
[11.1 Design a test of decisions, not a dramatic stor]
    |
    v
[11.2 Facilitator packet and release ledger]
    |
    v
[11.3 A decision log that a second shift can reconstr]
    |
    v
[11.4 Score the artifact, then discuss the decision]
    |
    v
[11.5 Run a negative control separately]
    |
    v
[Practice: supervised 70-minute tabletop]
    |
    v
[Subject synthesis]
    |
    v
[Worked example] --> [Exercise EX051]

Learning outcomes and connection

The learner can run a bounded tabletop rather than merely recite the CTN case. Given the service map, delayed case cards, a decision charter and an observer, the learner will produce time-stamped decisions, requests, stop conditions and handoffs. Chapter 10 supplied the draft P1 playbook; this chapter tests whether another reader can use it at the moment a decision is needed. Chapter 12 will turn observed failures into a portfolio and an improvement plan. Neither exercise authorizes access to a live system.

NIST SP 800-84 describes designing, conducting and evaluating IT exercises; it is a 2006 publication, so the CTN procedure and scoring below are original classroom adaptations, not a current mandatory NIST method. CISA's tabletop package documents distinguish planner, facilitator/evaluator and participant materials and include an after-action template. The practical lesson is to define objectives and observation criteria before revealing a scenario. A tabletop can reveal a missing owner or confusing handoff; it cannot prove that a control works in production. A functional test would require separate approval, isolation, safety checks and technical evidence.

11.1 Design a test of decisions, not a dramatic story

Define four observable objectives for this rehearsal: O1, preserve the time-sensitive IdP source through the authorized custodian; O2, select a proportionate containment option and keep approval, execution and verification separate; O3, stop a recovery candidate when a preannounced acceptance criterion fails; O4, communicate service status without turning an incomplete privacy assessment into a disclosure finding. For each objective specify the evidence a reviewer should see: a bounded request, an action record, a failed-test decision and two status lines. “Respond well” has no observable threshold and cannot be scored consistently.

Use three clocks. The case clock is the UTC time printed on C cards. The release clock tells the facilitator when a participant may see a card. The exercise clock is elapsed classroom time. A participant who receives C20 in minute 30 does not thereby place its event at 09:30 UTC. Record all three separately. This is especially important for C04's four-minute device offset, C10's conditional timezone conversion and C22's 08:52 event that is only released at 09:50. Backdating a decision with late evidence would make a poor plan look prescient.

The exercise is a discussion and paper-artifact rehearsal. The facilitator does not issue system commands, send messages, simulate actual account changes or invent missing logs. Give each role only the approved classroom packet. Use role cards for the analyst, incident lead, service owner, identity custodian, application/recovery custodian, communications/privacy reviewer and observer. One learner may play more than one role, but the recorder still names which decision right was used. The observer should not suggest the model answer while the team is deciding.

11.2 Facilitator packet and release ledger

The packet contains the public case cards as written in Chapters 1–9, the draft P1 specimen, a blank decision log, an objective sheet and a stop card. It contains no private contacts, personal records or production credentials. A useful release ledger for a 70-minute classroom exercise is below. The exercise-minute ranges are a facilitation plan; the listed UTC times remain the case's own sequence.

Lesson table; scroll horizontally to read all columns →

At each checkpoint the facilitator asks: “What did you know then, what action is proposed, who could approve it, what result would change your decision, and when will you review?” A participant may request information absent from the packet. The facilitator replies “not supplied” unless a scheduled card contains it; that answer becomes an observed evidence gap, not a fictional negative test. If the team asks to disable a live account, the facilitator points to the stop card and asks for a paper approval record and an alternative. If a participant prematurely uses a later card, freeze that statement, note the timing error and ask for a decision based only on released evidence.

11.3 A decision log that a second shift can reconstruct

A useful row contains exercise minute; case checkpoint UTC; card IDs available; observation; inference; option; approval status; operator/verification status; owner; next trigger. Keep the status fields separate: requested, approved, executed, checked and accepted are different events. A row for C17 at 09:37 can say “service owner approved S42 write hold to 09:47; operator execution pending.” Only after C18 may a later row say “activated 09:39.” C19's one-route denied edit does not certify all write routes. C21 is a new extension approval, not an extension inferred from C17. C24's task disablement does not remove C22's recovery method.

Worked example — the failed recovery gate. At exercise minute 45, the team sees C27. It has already written five acceptance criteria from Chapter 8: slot values, both write routes, new audit actor/request fields, preserved originals, and rollback copy/owner. The slot values and manual route pass, but the alternate-route actor field is blank. The recorder writes “isolated candidate no-go; criterion 3 failed; retain test artifact; application owner to correct mapping.” This is stronger than “4/5 pass” because the gate was conjunctive. At minute 49 C28 supplies a changed isolated candidate that passes its supplied checks; the team logs a new isolated result and waits for C29's distinct production approval. At minute 55 C30 supports limited service acceptance, not incident closure. The observer can reproduce the rationale without seeing an outcome chosen after the fact.

11.4 Score the artifact, then discuss the decision

Score each objective 0–2 against a preannounced artifact criterion: 0 absent or contradicted, 1 present but missing a time, source, owner or boundary, 2 complete enough that a second responder could reproduce the decision. The total is 0–8. For O1, a 2-point artifact names the C08 expiry, the 08:50–09:05 bounded window, identity custodian, permission and pending collection. For O2, it keeps C17/C18/C19/C21 distinct and a rollback/expiry visible. For O3, it stops C27 and states which exact criterion failed before considering C28. For O4, it separates C30's limited service return from C31–C33's unfinished external-delivery assessment and marks external text unsent until approved. A score says how well the paper decision was recorded, not whether CTN was secure.

Two observers may disagree. They should cite the exact row and criterion, record both provisional scores and resolve only after comparing the artifact with the rubric. If a role never received an inject because the facilitator skipped it, mark the objective not observed and report the conduct defect; do not score the participant zero for information never released. A late correct answer cannot repair an earlier unsafe action by erasing the old log. Preserve the sequence and annotate the correction. This is how the exercise measures process quality rather than rewarding hindsight.

11.5 Run a negative control separately

After the main case, reset the exercise clock and run Chapter 10's V10 and V10b as independent 8-minute branches. V10's correct slot and exact approved job record should lead to bounded documentation and an owner-led downgrade review, not a disruptive account action. V10b's unavailable approver before a log-expiry warning should lead to an authorized preservation escalation while disruption remains pending. Do not paste V10's correct-state observation into the main CTN case. Do not use main-case C25 to solve V10b's owner gap. A well-formed P1 playbook must support all three branches without silently acquiring more authority or evidence.

The stop card also has a learning purpose. If a learner proposes a broad tenant change without target, approval or rollback, pause the simulation, record the proposal verbatim, ask for its scope and compare it with the least disruptive alternative. The goal is correction with a durable artifact, not embarrassment. In a real organization a tabletop discovery becomes a readiness ticket, while any actual emergency response remains governed by the organization's approved procedures.

Audit the release boundary before scoring a decision

A good facilitator can accidentally make a weak response look strong by handing out the right evidence too soon. The learner then makes a seemingly well-supported decision that no responder at the stated checkpoint could have made. Avoid this by treating the release ledger as a test fixture. For every submitted decision, identify (1) its case checkpoint, (2) the actual exercise minute when it was written, (3) the set of cards the participant had received by that minute, and (4) the card release times. A card's event time may be earlier than its release time: C22 describes an 08:52 configuration change but is not available until 09:50. An event timestamp does not grant the earlier team access to that evidence.

The card list in a decision log is a claimed dependency set, not merely a bibliography. If a 09:42 decision cites C20, the argument has a temporal defect even if C20 later supports the same action. Preserve the submitted line, mark the defect and ask the learner to reconstruct a 09:42 decision using only C17–C19 and earlier released material. If the alternative remains defensible, record the revised rationale as a new exercise artifact. Do not silently edit the first line or give full credit for knowledge acquired later. This separates sound judgment under uncertainty from a fortunate outcome.

The following tiny, invented L11 audit is an offline exercise. It is not an access-control system or a replay of a production incident. It assumes the facilitator actually released each named card at the printed time; a real classroom must compare the script's table with its signed release log. The exercise-minute clock is omitted from the program because its only legitimate source is the observer's record, not a value that can be inferred from CTN's UTC cards.

from datetime import datetime

released = {
    "C13": "09:29", "C16": "09:35", "C17": "09:37",
    "C18": "09:39", "C19": "09:42", "C20": "09:46",
    "C27": "10:16", "C28": "10:20", "C29": "10:24",
    "C30": "10:45", "C33": "11:00",
}
decisions = [
    ("D1", "09:35", ["C13", "C16"]),
    ("D2", "09:42", ["C17", "C18", "C19"]),
    ("D3", "09:42", ["C20"]),
    ("D4", "10:20", ["C28", "C29"]),
    ("D5", "11:00", ["C30", "C33"]),
]

def minute(hhmm):
    parsed = datetime.strptime(hhmm, "%H:%M")
    return parsed.hour * 60 + parsed.minute

for name, checkpoint, citations in decisions:
    unseen = [card for card in citations if minute(released[card]) > minute(checkpoint)]
    if unseen:
        details = ", ".join(f"{card} at {released[card]}" for card in unseen)
        print(f"{name}: future card at {checkpoint}: {details}")
    else:
        print(f"{name}: release-order check passes at {checkpoint}")

Run with Python 3 in an empty local directory. Expected output is reproduced in the final worked-results section. No package installation, network, credential or real customer record is needed. Then modify a copy of D3 to cite C19 rather than C20. The line now passes the mechanical test, but ask a harder question: does C19's one-route denial support the proposed action's claimed all-route effectiveness? It does not. Passing a release check is necessary for temporal integrity, not sufficient for technical validity. Similarly, D4 must not treat C28's isolated pass as C29's later production approval even after its citation is repaired.

A second negative control tests the fixture itself. Remove C19 from the released dictionary without changing D2. The simple program will raise a KeyError; that is a missing release-log entry, not proof that C19 never existed. An operationally robust reviewer would classify that line as unverifiable and request the facilitator's release receipt. For a classroom run, restore the fixture after observing the failure. This teaches the distinction between an absent field and evidence of absence. The assessor should report how many decision rows were eligible for scoring, how many failed temporal review, and which were unassessable because of missing facilitator evidence. Never hide unassessable rows by shrinking the denominator without saying so.

Contrastive example. An incorrect observer says, “The team cited C20 and selected the S42 hold by 09:42, so it anticipated the cause and earns O2 = 2.” A correct observer says, “C20 was unavailable at 09:42. Score the original row as temporally defective; ask whether C13–C19 support a bounded hold, and review the revised row separately. C19 shows one route only.” The second report can explain why a decision was reasonable while still exposing a training-process failure. This matters at the professional level because the person maintaining the playbook must diagnose whether the defect belongs to the participant, the facilitator's release schedule, the playbook's decision fields or the observer's scoring practice. NIST SP 800-84 provides general guidance on designing, conducting and evaluating IT exercises; this release-audit fixture and its scoring rule are original CTN classroom material.

Common errors and limits

• Releasing C20 before the 09:42 hold check and then calling the earlier choice “obviously right.” Record the release violation and rerun that decision.
• Treating a role-played “yes” as an operational authorization. The role card permits only a classroom decision receipt.
• Scoring the quality of a vivid explanation while the submitted decision log lacks source IDs, owner, expiry or verification.
• Calling C30 a complete recovery or C33 proof of no external delivery. Both are bounded observations with open follow-up.
• Treating a tabletop's pass as a validated production control. Functional testing needs a separate, permitted test plan.

Practice: supervised 70-minute tabletop

Assign roles, print the release ledger and use only synthetic cards. Run the six checkpoints, collect the decision log, the C08 preservation request, the C27 gate record and two 11:00 status lines. The observer scores O1–O4 without seeing the answer appendix. Repeat the two negative-control branches. A second learner tries to reconstruct one 09:42 and one 11:00 decision from the submitted packet; record fields they cannot recover. Finally hold a ten-minute debrief that yields one strength, one failure, one unknown and one assigned retest for Chapter 12.

Exercises — subject 11

EX051 — Exercise control sheet. Inputs: the six release windows above and objectives O1–O4. Produce a one-page sheet with objective, inject/case-time, participant artifact, observer criterion, facilitator stop rule and reset rule. Use paper only. A 2-point result keeps the three clocks distinct and never releases a later card to justify an earlier decision.

L11 extension for EX051: Add a release-receipt column and a rule for a missing receipt. Explain why the card's event time cannot substitute for receipt time. Use the offline audit above as an optional check, then identify one decision that still needs semantic review after it passes the check.

EX052 — Time-gated decision log. Inputs: C16–C24. Produce five or more rows covering tolerance, initial approval, execution, one-route check, extension, task action and 10:07 expiry. Mark each row's source and status. A 2-point result does not turn C16 into approval, C19 into full-route validation or C24 into complete eradication.

L11 extension for EX052: Run D1–D5, name the two future-card errors and rewrite the 09:42 line without C20. Keep the corrected line as a new version rather than overwriting the submitted one.

EX053 — Recovery gate replay. Inputs: C26–C30 and the five criteria in §8.4. Write two distinct isolated-test decisions and one production/monitoring handoff. Identify the first failed C27 criterion and the approval/verification boundary between C28 and C29. A 2-point result refuses a whole-snapshot restore that would overwrite S40/S41.

EX054 — Observer calibration. Input: a fictional participant artifact that correctly requests C08 preservation, records C17 approval but omits C18/C19, says “C27 failed audit,” and drafts a learner message saying “12 people exposed.” Score O1–O4 as 0, 1 or 2, with one citation per score and a correction request. A 2-point result does not reward the unsupported exposure claim.

EX055 — Three-branch handoff. Inputs: main CTN through C33, V10 and V10b. Prepare a three-row handoff with branch-specific evidence, action status, owner, stop/review trigger and one source that would change the decision. A 2-point result keeps the variants separate and leaves V10b disruption unexecuted.

Subject synthesis

A supervised simulation tests decision traceability under staged information. The CTN cards can reveal whether P1 helps a second responder choose, stop and hand off; they cannot demonstrate control effectiveness outside this fictional classroom. Chapter 12 will collect the decision artifacts, test their completeness and assign improvements with owners and retest criteria.