Chapter 12 — Case portfolio and after-action review

Lesson map

Lesson 12 published. Map, example, EX056.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["12.1 Portfolio structure: claims need an evidence pa"]
  N2["12.2 Defend the decision at its checkpoint"]
  N3["12.3 Three cases in one comparative portfolio"]
  N4["12.4 After-action review: from observation to retest"]
  N5["12.6 Final practical defense and assessment"]
  N6["Practice: portfolio defense"]
  N7["Subject synthesis and next step"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX056"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["12.1 Portfolio structure: claims need an evidence pa"]
  N2["12.2 Defend the decision at its checkpoint"]
  N3["12.3 Three cases in one comparative portfolio"]
  N4["12.4 After-action review: from observation to retest"]
  N5["12.6 Final practical defense and assessment"]
  N6["Practice: portfolio defense"]
  N7["Subject synthesis and next step"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX056"]

[Learning outcomes and connection]
    |
    v
[12.1 Portfolio structure: claims need an evidence pa]
    |
    v
[12.2 Defend the decision at its checkpoint]
    |
    v
[12.3 Three cases in one comparative portfolio]
    |
    v
[12.4 After-action review: from observation to retest]
    |
    v
[12.6 Final practical defense and assessment]
    |
    v
[Practice: portfolio defense]
    |
    v
[Subject synthesis and next step]
    |
    v
[Worked example] --> [Exercise EX056]

Learning outcomes and connection

The learner can assemble a source-traceable portfolio from Chapters 1–11, defend a choice at the time it was made, distinguish an observed weakness from an unproven root-cause story, and create an after-action improvement record with a specific owner, due checkpoint and falsifiable closure test. This is the final subject of the professional course. A portfolio makes learning assessable; it is not a professional certification, forensic opinion or declaration that CTN's fictional incident is fully closed.

NIST SP 800-184 supports recovery planning and learning from events, while NIST SP 800-84 and CISA's tabletop package documents support exercise evaluation and improvement planning. The specific CTN portfolio, scoring and deadlines below are original teaching choices. A real after-action report must follow the organization's own governance and privacy rules.

12.1 Portfolio structure: claims need an evidence path

Submit five linked sections. A. Service and authority contains the C01 dependency map and the Chapter 2 action charter, with blank alternates left visible. B. Intake and evidence holds the 09:14 priority record, C09/C10 receipts and a source-traceable timeline. C. Action and recovery holds the C17–C30 decision chain, including the C27 no-go and C28 revised isolated test. D. Communication holds separate C30 service and C31–C33 data-assessment lines and the unsent approval request. E. Exercise and improvement holds the Chapter 11 release ledger, observer record, branch comparison and after-action plan. Each item needs a source ID, time basis, authoring role, status and link to the exact chapter or card; a bare screenshot or unexplained score is not a portfolio.

Use an evidence index rather than duplicating every card. A concise row reads: Claim: S42 version 19 unavailable at 09:29; Source: C13 read-only query; Acquired/released: 09:29; Status: observed stored state; Limit: no actor or user-impact count; Used by: containment option review. The same index should record a negative claim: “C09 has no supplied original object digest; therefore this portfolio does not certify full export integrity.” A negative claim states what has not been established, not that the opposite is true. Include a version number: if the gateway gap is later filled, keep the 11:00 assessment and append a new one rather than rewrite history.

12.2 Defend the decision at its checkpoint

Hindsight can make any earlier decision look careless or heroic. Reviewers should ask: “What was available at 09:14?” At that point C08 gave a 10:00 source-expiry warning, but C09 had not been received and C13 had not yet verified stored state. An urgent bounded preservation request was defensible. An account-wide disablement would have required a different harm/authority basis. At 09:35 C13 showed a wrong authoritative value, C14 weakened the approved-job explanation and C16 gave ten-minute tolerance; a proposal for an S42 hold became more defensible, but C17 approval was still in the future. At 10:16 C27's audit field failed the gate; at 10:20 C28 supplied a corrected isolated result, but not production approval. Defending these transitions shows professional judgment better than repeating the final narrative.

For each defended choice use a four-part argument: available observations; at least one live alternative; expected harm and service cost of options; next observation that would change the choice. Then attach the owner and approval status. A reviewer can disagree with an option while accepting the reasoning as defensible if it honors the evidence and authority available then. Conversely, an outcome that happened to work does not rescue an undocumented or unauthorized decision.

12.3 Three cases in one comparative portfolio

The main CTN path, V10 and V10b are three separate classroom cases. The main path shows an unapproved task associated with an S42 edit and a separate unresolved identity change. V10 is a negative control with a correct slot and an exact approved job record in its supplied scope. V10b asks what to do when an approver is unavailable before a stated expiry. They test different playbook branches. They are not three observations of a single real incident.

Lesson table; scroll horizontally to read all columns →

Cross-case comparison should reveal the invariant: preserve provenance and authority while varying the action with evidence. It should not flatten every case into “contain first.” A tabletop facilitator may add a future variant, but it must have its own label and cannot silently repair CTN's source gaps. This is also why a playbook score of 12/12 for document design in Chapter 10 is only a hypothesis about usability until independent readers run these branches.

12.4 After-action review: from observation to retest

An after-action record needs objective, observed result, supporting artifact, limitation, possible contributing condition, corrective action, owner, due checkpoint and closure test. Avoid a blame conclusion from an account label or empty actor field. For example, observed: “The C01 after-hours identity alternate is blank; V10b participants had no supplied approver.” Contributing condition hypothesis: “ownership coverage was not defined in the exercise charter.” Improvement: “incident lead and identity owner designate a real alternate in the actual organization, obtain that role's acceptance and rehearse a call path.” Closure test: “in a later approved drill, the alternate receives a scoped request before a preannounced expiry and the observer records receipt and decision right.” Merely writing a name in a diagram would not satisfy the test.

Another row concerns C27: the first isolated correction passed visible data but failed the alternate-route audit actor field. Improvement is not “ignore audit to meet time target.” The application owner should own a mapping fix, the reviewer should rerun both routes with a negative control, and release should stop on another blank field. A third row concerns C09: missing export-object metadata lowers confidence in a preserved copy. The identity custodian should define a bounded collection receipt with source object ID, byte count, digest, collector, time and authority; then a permitted rehearsal should produce that receipt. A fourth row keeps C33's gateway gap as an assigned evidence request, not a finding of delivery or nondelivery. Each row needs a deadline appropriate to the organization; this classroom uses “before the next drill” because it has no real operational calendar.

12.5 Metrics with denominators and limits

One useful measure is decision-record completeness: among the five tabletop action proposals selected for review, how many contain target, evidence, approver status, expiry/stop, verification and owner? If four meet all six fields, report 4/5 = 80% in this rehearsal, with the omitted field named. Do not report 80% of CTN's live decisions or infer future control reliability. A second measure is gate discipline: of two isolated recovery candidates C27/C28, one failed a preannounced criterion and was stopped; record 1/1 failed candidates stopped, not “100% recovery success.” A third is source preservation timing: C08 stated 10:00 expiry and C09 arrived 09:18 after a 09:16 approval; a learner can measure those case times, but the packet lacks enough detail to certify the full export's byte integrity. Report both the observed interval and the missing provenance field.

Metrics can reward the wrong behavior if a denominator is omitted. “Average time to containment” encourages a fast but unjustified account disablement; “percentage of alerts closed” can reward premature closure. Pair speed with safety and quality: authorized request before a verified source deadline; changes with exact approval and expiry; failed acceptance gates stopped; and case claims whose source and confidence are stated. Sample sizes in a tabletop are small. Treat a good score as a prompt for a larger supervised retest, not statistical proof.

Decide whether an improvement is closed, rather than merely assigned

An after-action ticket is a hypothesis about a fix. It becomes credible when a subsequent, independently observable test challenges the original failure. Separate at least five states in the portfolio: observed defect, action assigned, change reported, retest observed, and reviewed closure candidate. A ticket with an owner and deadline is only assigned. A merged playbook change is only a reported change. A passing rehearsal for one route may leave another route open. An independent reviewer can accept a classroom closure candidate while reserving production release for the actual organization's approval process. This vocabulary prevents a neat task board from masquerading as recovered capability.

Consider the C01 alternate-owner gap. A planned repair might name an after-hours substitute in a contact table. The test must do more: in a permitted rehearsal, a facilitator sends a scoped preservation decision request at a known exercise minute, records whether the alternate receives and understands it before a declared deadline, and checks that the alternate can exercise the relevant decision right. A contact who answers the phone but cannot approve the action does not close the authority gap. If the primary and alternate both know the answer because they attended the design meeting, repeat the test with an unbriefed duty holder or a changed shift. The objective is transferable process behavior, not one person's memory.

For C09, a future export-receipt template can require source object ID, selection window, collector, authority, byte count and digest. Those fields would improve a new exercise receipt. They cannot reconstruct the original missing object digest of C09 by inference. Keep C09's historical limit in the portfolio even if a later drill produces a complete receipt. The test should include a deliberately altered local copy that fails digest comparison and a missing-field control that is rejected as incomplete. This checks both the positive route and the condition that caused the earlier uncertainty.

For C27, test the corrected mapping on both relevant write routes, inspect the new actor and request fields, preserve the failed first result, and ask whether a route not exercised could still create an unattributed change. A single dashboard green state is not enough. For C33, distinguish two improvement targets: resolving the historical 10:49–10:51 gateway question, if authorized sources still exist, and making future gateway coverage observable. A successful future retention rehearsal cannot prove what happened in the earlier missing window. Conversely, an authenticated historical delivery receipt would update the privacy assessment without proving that the monitoring fix now works. These require separate evidence paths and owners.

The following P12 fixture is a standalone, invented portfolio-quality exercise, not an update to the CTN case. Its four ticket rows are examples of what a learner might submit after a later hypothetical rehearsal. They contain no person, address, tenant or live credential. The rule deliberately checks only whether there is enough classroom evidence to consider closure; it neither approves a real change nor authenticates the supplied evidence. The reviewer must first verify that the evidence refers to the named test and its stated scope.

tickets = [
    {"id": "I1", "owner": "exercise lead", "test": "alternate receipt before deadline",
     "result": "pass", "evidence": "signed drill row 7", "reviewer": "independent observer"},
    {"id": "I2", "owner": "identity custodian", "test": "export receipt negative control",
     "result": "", "evidence": "", "reviewer": "independent observer"},
    {"id": "I3", "owner": "application owner", "test": "both routes carry actor and request",
     "result": "fail", "evidence": "alternate route blank", "reviewer": "independent observer"},
    {"id": "I4", "owner": "gateway custodian", "test": "future gap alarm and receipt",
     "result": "pass", "evidence": "isolated replay 4", "reviewer": ""},
]

for ticket in tickets:
    if not ticket["owner"] or not ticket["test"] or not ticket["result"] or not ticket["evidence"]:
        verdict = "open: missing retest evidence"
    elif ticket["result"] != "pass":
        verdict = "open: failed criterion"
    elif not ticket["reviewer"]:
        verdict = "open: independent review pending"
    else:
        verdict = "classroom closure candidate"
    print(f"{ticket['id']}: {verdict}")

Run this with Python 3 offline; the exact four lines appear in the worked-results section. Then make two separate copies. In one, change I1's result to fail: the owner and signed row still exist, but the verdict must become open. In the other, fill I4's reviewer field: the mechanical result becomes a closure candidate, yet the human reviewer must still ask whether the isolated replay represents the production gateway route and whether the historical C33 question remains unresolved. This is why the code says candidate. A field-completeness program cannot judge sampling, authenticity, safe scope or whether a measured improvement addresses the original defect.

Contrastive review. The weak portfolio says, “All four improvements are complete; they have owners and the playbook was revised.” The stronger portfolio says, “One later classroom alternate-owner drill has a reviewable pass; the export-receipt rehearsal has no result, the alternate-route audit test failed, and the gateway test lacks independent review. None changes the historical C09 or C33 uncertainty. The next owner action, evidence request and due checkpoint remain explicit.” The stronger statement is longer because it reports the state of each claim, not because it inflates the measured progress. NIST SP 800-84 covers exercise evaluation generally; P12 and its closure logic are original classroom examples.

12.6 Final practical defense and assessment

The learner submits the index, three branch artifacts and an after-action plan. A reviewer selects one early and one late checkpoint without warning. The learner has three minutes each to reconstruct available evidence, decision status, missing source and next test. Then the reviewer presents a counterfactual: “Suppose the gateway gap later contains an authenticated delivery receipt without recipient mapping.” A defensible response versions the privacy assessment, requests owner-approved mapping and communication review, preserves the older 11:00 note, and does not declare 12 affected people from 12 tokens. No actual notice is sent.

Use a five-criterion 0–2 assessment: source/time traceability; authority and action status; alternative reasoning; test/rollback quality; improvement ownership. A 2 requires a concrete artifact and stated limit, 1 a plausible but incomplete artifact, 0 a missing or contradicted one. A total of 8/10 may be a classroom pass only if neither source/time nor authority receives 0; a high total cannot offset fabricated evidence or an unauthorized action. Record the assessor's comments and a reattempt requirement when either safety-critical criterion fails. This rubric evaluates this course's portfolio; it is not a credential or claim that the manuscript has passed the book publication gate.

Common errors and limits

• Presenting a polished chronological story with no source or release-time citations. Rebuild the evidence index from original cards.
• Calling the unapproved task the human root cause. C20/C23 support a mechanism and approval classification, not an actor or motive.
• Treating C30's limited service return as privacy closure. Maintain two tracks and the C33 follow-up.
• Closing an improvement because its ticket exists. Require an owned, observed retest against the specified failure.
• Describing a 70-minute classroom tabletop as 70 minutes of real response capability. The setting and denominator are different.

Practice: portfolio defense

With two learners, exchange only the portfolio index and linked artifacts. The reviewer chooses a claimed C09 integrity statement, a C17–C21 hold statement and a C33 communication statement; ask the author to navigate to sources and name what each cannot prove. Run the V10 and V10b branch defense. Then score the five criteria and write an improvement row for every missing safety-critical field. No personal data, live tenant or production notification is involved.

Exercises — subject 12

EX056 — Evidence index. Inputs: C01–C33 and your Chapter 11 artifacts. Create eight or more source-traceable claim rows covering readiness, intake, preservation, containment, recovery and communication. For each give case/release time, status and one prohibited inference. A 2-point result includes C09's provenance gap and C33's gateway gap.

EX057 — Checkpoint defense. Inputs: 09:14 C08, 09:35 C13–C16, 10:16 C27 and 10:20 C28. Write a four-row decision table with available observations, alternative, owner/approval, action status and next discriminator. A 2-point result never imports a later card into an earlier row.

EX058 — Comparative cases. Inputs: main CTN, V10 and V10b. Produce three branch records with trigger, evidence, safe decision, owner, limitation and review/closure condition. A 2-point result identifies the same provenance/authority invariant without treating the cases as one incident.

EX059 — After-action plan. Inputs: C01, C09, C27, C33 and the Chapter 11 observer notes. Produce four improvement rows with observed finding, source, contributing-condition hypothesis, owner, due checkpoint and an independent retest that can fail. A 2-point result does not claim a root cause or a control repair from an assigned ticket alone.

P12 extension for EX059: Run the offline ticket fixture. State the evidence needed to promote each open row, and explain why a new complete export receipt cannot retroactively supply C09's absent source-object digest.

EX060 — Metrics and oral defense. Inputs: the 4-of-5 complete classroom action records, C27/C28 gate outcomes and the counterfactual receipt in §12.6. Calculate two correctly bounded metrics, then give a 150–220-word defense of what changes in service and privacy status. A 2-point result states denominators, preserves the old 11:00 note and refuses to equate tokens with people.

P12 extension for EX060: Report the fixture's closure-candidate ratio with its denominator and classroom-only scope. Do not turn that ratio into a production success measure or into a finding about C33's historical delivery gap.

Subject synthesis and next step

This twelve-subject draft now spans readiness, authorized response, evidence, competing explanations, containment, cause-path verification, recovery, communication, a reviewed playbook, a supervised simulation and a defensible portfolio. The 60 exercise-answer pairs support practice, but the manuscript is still short of the required verified 300 A4 pages and has not passed independent technical, editorial or accessibility review. The next learning level's technical course will demand deeper system modeling, reproducible labs and more explicit failure/rollback evidence. The next editorial task for this volume is substantial expansion and review, not relabeling the draft complete.