Chapter 24 — Applied methods: compare two incident playbooks

Chapter 24 — Applied methods: compare two incident playbooks

Lesson map

Lesson 24 published. Map, example, EX070.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["24.1 What a playbook is"]
  N2["24.2 Comparison criteria"]
  N3["24.3 Testing by walkthrough"]
  N4["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4
  N4 --> EX["Worked example"]
  EX --> Q["Exercise EX070"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["24.1 What a playbook is"]
  N2["24.2 Comparison criteria"]
  N3["24.3 Testing by walkthrough"]
  N4["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX070"]

[Learning outcomes and connection]
    |
    v
[24.1 What a playbook is]
    |
    v
[24.2 Comparison criteria]
    |
    v
[24.3 Testing by walkthrough]
    |
    v
[Chapter synthesis]
    |
    v
[Worked example] --> [Exercise EX070]

Learning outcomes and connection

You will be able to (1) compare two playbooks for the same scenario against explicit criteria; (2) test them with a tabletop walkthrough using the same inject; and (3) recommend a merged or chosen playbook with evidence. Chapters 18–23 built and measured actions; playbooks package them for a scenario.

24.1 What a playbook is

A playbook is a scenario-level guide: when it applies, roles, decision points, the runbooks it invokes, communications and closure criteria. A runbook is a step-level procedure for one task. CISA's federal incident response playbook is a public example of a scenario-level structure; many organizations also publish or share sector playbooks. A playbook should be short enough to use under stress and precise at decision points. Fernmoor's business email compromise comparison showed the difference: the 14-page vendor template covered every email threat but named no runbooks, while the 3-page local playbook did.

24.2 Comparison criteria

Fernmoor's criteria: (1) trigger clarity — can the duty analyst tell when it applies? (2) decision points — are decisions, owners and authority explicit? (3) evidence preservation before change; (4) actionability — does each step name a runbook or command? (5) communications — who is told, when, by whom? (6) closure — measurable conditions; (7) fit — does it match Fernmoor's tools and charter? Score each 0–2 with a written reason; scores are structured judgements, not measurements. In the BEC comparison, these scores separated two quite different strengths: the vendor template scored higher on evidence preservation and communications, while the local playbook scored higher on decisions, actionability and fit. Reading the criteria one by one led to a merged playbook rather than a simple winner.

24.3 Testing by walkthrough

Read both playbooks against the same scripted inject with two different teams (or the same team on different days, noting learning effects). Time to first containment decision, number of questions the playbook could not answer, and errors made are recorded. In Fernmoor's BEC walkthrough, the team using the local playbook reached the first containment decision in 12 minutes against 31, but forgot to contact the bank about recalling the payment. Recording that omission was as important as the timing, because a recall is the most time-critical step in real cases. A later exercise is planned with counterbalanced teams.

24.4 Worked example (synthetic): business email compromise

Fictional. Playbook A is a vendor template (14 pages); Playbook B was written by Fernmoor's Response Desk (3 pages linking runbooks). Inject: "Finance reports an invoice payment changed to a new bank account after an email from a known supplier."

Walkthrough: team using B reached the first containment decision in 12 minutes versus 31 with A, but forgot to contact the bank about recalling the payment — the most time-critical step in real BEC cases. Recommendation: adopt B, import A's evidence-preservation step and its bank/supplier communications section, and add measurable closure conditions (payment recall status, mailbox rules removed, supplier notified, verification of supplier's legitimate contact).

24.5 Common errors and limits

• Picking the longer playbook because it looks thorough.
• Testing two playbooks on the same team back-to-back without noting learning effects.
• Treating total scores as precise; read the criterion-level gaps.
• Playbooks that duplicate runbook detail and go stale.

The BEC comparison illustrates three of these. The vendor template was longer and looked thorough but scored lower overall; the totals of 7 and 11 hid that the shorter playbook missed bank contact; and the merged version took only the vendor template's evidence-preservation step and communications contacts, leaving runbook detail in the runbooks.

Exercises — chapter 24

Attempt each exercise before opening the worked results. Every case, organization, person, address and number in an exercise is synthetic unless a cited public source is named.

EX070 — Scoring reasons. Steps: write one-sentence reasons for B's scores on evidence-first and communications. Rubric: 3 = specific, refer to text omissions; 2 = generic; 1 = score only; 0 = none.

EX071 — Closure conditions. Steps: write five measurable closure conditions for the merged BEC playbook. Rubric: 3 = five measurable (recall outcome recorded, rules removed and verified, sessions revoked, supplier contacted via verified channel, finance procedure change logged); 2 = three; 1 = vague; 0 = none.

EX072 — Test design. Steps: design a fairer comparison than the business email compromise walkthrough. Rubric: 3 = two comparable teams or counterbalanced order, same inject, measures defined in advance, observer, noted limits; 2 = partial; 1 = vague; 0 = none.

Chapter synthesis

Compare playbooks with explicit criteria, scored with reasons, and test them by walkthrough on the same inject. Combine the strengths of each into a short, actionable playbook with measurable closure conditions.

Continue studying — chapter 24 of 30

← Chapter 23 · Course chapter index · Chapter 25 →

Continue to the next authored lesson.