Chapter 11 — Applied project: redesign an incident program
Lesson map
Lesson 11 published. Map, example, EX051.
flowchart TD
N0["Learning outcomes and connection"]
N1["11.1 Gap assessment"]
N2["11.2 Prioritization"]
N3["11.3 Design options"]
N4["11.4 Piloting"]
N5["11.5 Evaluation"]
N6["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX051"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["11.1 Gap assessment"]
N2["11.2 Prioritization"]
N3["11.3 Design options"]
N4["11.4 Piloting"]
N5["11.5 Evaluation"]
N6["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX051"]
[Learning outcomes and connection]
|
v
[11.1 Gap assessment]
|
v
[11.2 Prioritization]
|
v
[11.3 Design options]
|
v
[11.4 Piloting]
|
v
[11.5 Evaluation]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX051]
Learning outcomes and connection
You will be able to (1) conduct a gap assessment of an incident program using evidence; (2) generate and compare design options; (3) pilot one change with an evaluation plan; (4) evaluate the pilot against explicit criteria; and (5) involve stakeholders throughout. This project integrates the whole book: decision framing (1), command (2), evidence (3, 8), ethics and law (4, 7), containment and recovery (5, 6), leadership (9) and research reading (10). Chapter 12 transfers the results into the organization.
11.1 Gap assessment
Assess the program against a reference (NIST SP 800-61 Rev. 3 and CSF 2.0 outcomes; the FIRST CSIRT Services Framework for service scope) and against its own incident data. Evidence sources: case records, after-action reports, interviews, exercise results, metrics. A gap is a difference between required and actual capability, backed by evidence, with an impact statement.
Calder's assessment found two gaps this way: after-action reports cited handoff delays in 9 of 14 major incidents, and case records were incomplete in 40% of severity-2 cases. Each is a measured difference between what the charter requires and what happened.
11.2 Prioritization
Prioritize gaps by impact, frequency and feasibility. Recurring findings across after-action reports are strong signals; one dramatic incident is a weaker one.
Calder's handoff gap qualified as a strong signal on all three counts: it recurred in 9 of 14 after-action reports, it delayed the incoming lead's first action on live cases, and the cheapest option, a template with read-back, could be piloted in weeks. A single spectacular failure in one division would have been weaker evidence, however memorable. The 40% incomplete case records were linked to the same handoffs, so one change might address both gaps.
11.3 Design options
For each priority gap, generate at least two options with different cost and disruption profiles (process change, tooling, staffing, training, contractual). Evaluate them with the multi-criteria method used at bachelor's level, weights agreed with stakeholders.
Calder's options were a handoff template with read-back (process, low cost), a case-management workflow with required fields and reminders (tooling, medium cost), or both. The template was piloted first because it could be tested in weeks, while the workflow became a budget item for the next cycle.
11.4 Piloting
Choose one change to pilot in a bounded part of the organization for a defined period, with criteria fixed in advance, a comparison (another division or the pre-period, ideally both), data collection plan and stop rules.
Calder's pilot met each requirement: option A in the Logistics division for 8 weeks, with Retail as a concurrent comparison; criteria fixed in advance, a 20% fall in median minutes to first action, 85% record completeness and no increase in time to containment; and data from case-system events. Choosing one change made the result attributable.
11.5 Evaluation
Evaluate against the criteria; include process measures (was the change used as designed?), outcome measures, and unintended effects. Report uncertainty; small pilots rarely give precise effects.
Calder's result shows why the comparison and the caveats matter. Median minutes to first action fell 36% in Logistics and 5% in Retail over the same weeks, and completeness reached 87%. But with 10 to 12 handoffs per period, staff aware they were in a pilot, and a template that also prompts record completion, the effect is an estimate, and time to containment still had to be checked before rollout.
11.6 Worked example and lab (synthetic): handoff delays
Fictional. Calder's after-action reports from the past year show handoff delays in 9 of 14 major incidents; case records are incomplete in 40% of severity-2 cases. Gap: no standard handoff or minimum case-record content. Options: (A) handoff template plus read-back (process; low cost); (B) case-management workflow with required fields and automated reminders (tooling; medium cost); (C) both. Pilot: option A in the Logistics division for 8 weeks; Retail as a concurrent comparison. Criteria: median minutes from shift change to the incoming lead's first action on the case falls by at least 20%; case-record completeness (required fields present at closure) reaches 85%; no increase in time to containment.
Save as lab11.py.
# Lab PG-11 - evaluate a handoff pilot with a concurrent comparison (synthetic, stdlib only)
from statistics import median
# minutes from shift change to incoming lead's first action, per handoff
LOG_PRE = [42, 35, 50, 38, 61, 44, 39, 47, 55, 40]
LOG_POST = [25, 31, 22, 40, 28, 19, 33, 27, 36, 24, 30, 26]
RET_PRE = [45, 37, 52, 41, 48, 39, 58, 43]
RET_POST = [44, 40, 39, 47, 51, 38, 42, 46, 41]
COMPLETE = {"Logistics": (47, 54), "Retail": (31, 49)} # (complete records, closed cases) in pilot weeks
def pct(a, b): return 100 * (b - a) / a
lp, lq, rp, rq = map(median, (LOG_PRE, LOG_POST, RET_PRE, RET_POST))
print(f"Logistics median {lp} -> {lq} ({pct(lp, lq):+.0f}%)")
print(f"Retail median {rp} -> {rq} ({pct(rp, rq):+.0f}%)")
print(f"difference in change: {pct(lp, lq) - pct(rp, rq):+.0f} percentage points")
for d, (c, n) in COMPLETE.items():
print(f"{d} completeness {c}/{n} = {100*c/n:.0f}%")
Expected output:
Logistics median 43.0 -> 27.5 (-36%)
Retail median 44.0 -> 42 (-5%)
difference in change: -32 percentage points
Logistics completeness 47/54 = 87%
Retail completeness 31/49 = 63%
Interpretation: in Logistics, median minutes to first action fell 36% while Retail fell 5% over the same weeks; the difference (about 32 points) suggests most of the improvement is associated with the pilot rather than seasonal factors. Completeness reached 87% against an 85% criterion. Caveats: 10–12 handoffs per period are few; divisions differ; staff knew they were in a pilot; the template also prompts record completion, so the two outcomes are not independent. Time to containment (not shown) must be checked for adverse effects before rollout. Recommendation: extend the template group-wide with option B scoped for the next budget cycle, and continue measurement.
11.7 Common errors and limits
- Gaps asserted without evidence.
- Piloting several changes at once, so effects cannot be separated.
- Criteria set after results.
- Declaring success from small samples without a comparison.
Calder's pilot avoided each of these errors. The two gaps came from after-action reports and case records, not from opinion. Only option A was piloted, with B deferred to the budget cycle. The 20% and 85% criteria were set before the eight weeks began. Retail served as a concurrent comparison, which separated the pilot's effect from seasonal change. Even so, the small number of handoffs is reported as a limit, and rollout includes continued measurement.
Exercises — chapter 11
Attempt each exercise before opening the worked results. Every case, organization, person, address and number in an exercise is synthetic unless a cited public source is named.
EX051 — Gap statement. Steps: write two evidence-backed gap statements for Calder with impact. Rubric: 3 = requirement, evidence (sources, counts), impact, owner; 2 = no evidence counts; 1 = opinion; 0 = none.
EX052 — Option comparison. Steps: compare A, B and C on cost, time to implement, disruption, expected effect and evidence strength; recommend a pilot. Rubric: 3 = table and justified pilot choice (A first because cheap, fast, testable); 2 = recommendation without table; 1 = preference; 0 = none.
EX053 — Run and critique Lab PG-11. Steps: run the lab and list four threats to the conclusion. Rubric: 3 = output reported; small n, Hawthorne effect, non-equivalent divisions, correlated outcomes, regression to the mean; 2 = two threats; 1 = not run; 0 = none.
EX054 — Stop rules. Steps: write stop rules for the pilot. Rubric: 3 = adverse effects (containment time increase beyond threshold, staff burden complaints), data-quality failure, incident severity override; 2 = one; 1 = vague; 0 = none.
EX055 — Stakeholder plan. Steps: identify stakeholders and their involvement at each project stage. Rubric: 3 = division leads, SOC, service owners, HR (for shift practices), executives; involvement at assessment, design, pilot, evaluation; 2 = list only; 1 = security only; 0 = none.
Chapter synthesis
Redesign an incident program from evidence: assess gaps with sources, prioritize recurrent high-impact gaps, compare options, pilot one change with pre-set criteria and a concurrent comparison, evaluate honestly with stop rules, and involve stakeholders at every stage.
Continue studying — chapter 11 of 12
← Chapter 10 · Course chapter index · Chapter 12 →
Continue to the next authored lesson.