Chapter 33 — Research methods: incident outcome measurement | Bachelor's Incident response and cyber resilience

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 33 published. Map, example, EX097.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["33.1 Outcomes and estimands"]
  N2["33.2 Comparison groups"]
  N3["33.3 Selection bias"]
  N4["33.4 Data quality"]
  N5["33.5 Pre-specification and robustness"]
  N6["33.6 Ethics"]
  N7["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX097"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["33.1 Outcomes and estimands"]
  N2["33.2 Comparison groups"]
  N3["33.3 Selection bias"]
  N4["33.4 Data quality"]
  N5["33.5 Pre-specification and robustness"]
  N6["33.6 Ethics"]
  N7["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX097"]

[Learning outcomes and connection]
    |
    v
[33.1 Outcomes and estimands]
    |
    v
[33.2 Comparison groups]
    |
    v
[33.3 Selection bias]
    |
    v
[33.4 Data quality]
    |
    v
[33.5 Pre-specification and robustness]
    |
    v
[33.6 Ethics]
    |
    v
[Chapter synthesis]
    |
    v
[Worked example] --> [Exercise EX097]

Learning outcomes and connection

You will be able to (1) define an outcome and an estimand for a response study; (2) choose a comparison group and recognize selection bias; (3) assess data quality before analysis; (4) specify exclusions and robustness checks in advance; and (5) state conclusions without overstating causality. Chapter 18 introduced study design; Chapters 12 and 15 gave statistics and measurement. This chapter applies them to the common question "did the new playbook improve outcomes?".

33.1 Outcomes and estimands

An outcome is a measured variable per case (for example hours to containment, or whether data loss occurred). An estimand is the exact quantity you intend to estimate: "the difference in median hours to containment between severity-2 phishing cases handled under playbook B and those handled under playbook A, among cases opened in 2026". Writing the estimand first prevents switching measures after seeing results.

In Lab 33 the estimand is the difference between East's change and West's change in monthly median containment hours, comparing the three months before East adopted playbook B with the three after. Writing it that way commits you to West as the comparison and to medians as the measure before the numbers are seen. A different estimand, such as the mean or a single post month, could give a different answer; that is why it is fixed first.

33.2 Comparison groups

Before/after comparisons confound the playbook with everything else that changed over time. Better options when randomization is impossible: concurrent comparison (some teams adopt first), interrupted time series with several pre- and post-periods, or difference-in-differences against a group that did not change. Each has assumptions you must state.

Lab 33 uses difference-in-differences: West did not adopt the playbook and serves as the comparison. Its key assumption is that, without the playbook, East and West would have followed parallel trends. The pre-period gap narrowed slightly from month to month, so the assumption is not exactly met, and the report should say how that affects the estimate. Without West, the naive before-and-after change would have credited the playbook with improvement that both teams shared.

33.3 Selection bias

If only some cases are handled with the new playbook (for example, analysts choose it for easy cases), the comparison is biased. Assignment rules and their adherence must be recorded.

The exercise on selection shows the risk in Lab 33's setting: if East analysts could choose playbook B case by case, they might use it for simpler cases, making it look better than it is. Assigning the playbook to a whole team or to blocks of time, and analysing by team regardless of use, avoids that bias.

33.4 Data quality

Check completeness, consistency of definitions over time, timestamp sources and label drift (Chapters 15 and 29). Report the share of cases excluded and why.

For Lab 33, that means confirming that East and West defined containment the same way and recorded it from the same system over all six months. If one team changed its form during the period, as in the response-time lab earlier, the comparison could reflect recording rather than performance. Report how many cases each monthly median rests on and how many were excluded for missing times. A medians-only table can hide a month where few cases were recorded.

33.5 Pre-specification and robustness

Write exclusions (for example, merged cases, test cases), the primary analysis and planned robustness checks (alternative outcome definition, excluding a holiday month, handling missing values) before analysis. Robustness checks show whether the conclusion depends on arbitrary choices.

Lab 33's robustness check, excluding the transition month, gives a similar estimate, which suggests the result does not hinge on how month 4 is treated. That check was meaningful because it could have changed the answer. List each check before analysis and report every one, including any that weaken the conclusion, rather than only those that agree with it.

33.6 Ethics

Outcome studies use case records containing personal data and analyst performance data. Minimize and pseudonymize; report aggregates; obtain approval where required (Chapter 18).

In Lab 33 the data are monthly medians per team, which reveal little about individuals. A case-level analysis, which the lab notes would be needed for an uncertainty interval, would bring personal data and analyst performance back into scope: pseudonymize analysts and patients, keep the key separately, and report only aggregates. Make clear to the teams that the study evaluates a playbook, not people, and keep results away from individual performance reviews. Where the case records come from real incidents rather than synthetic ones, seek approval before analysis begins, as the research-methods chapter requires.

33.7 Worked example and lab (synthetic): before and after a playbook

Save as lab33.py.

# Lab 33 - before/after vs difference-in-differences for a playbook change (synthetic, stdlib only)
from statistics import median

# median hours to containment per month; Team East adopted playbook B in month 4, Team West did not
EAST = [10.0, 9.6, 9.4, 8.0, 7.8, 7.5]
WEST = [11.0, 10.5, 10.2, 9.6, 9.2, 9.0]

def mean(xs): return sum(xs) / len(xs)
pre, post = slice(0, 3), slice(3, 6)
east_change = mean(EAST[post]) - mean(EAST[pre])
west_change = mean(WEST[post]) - mean(WEST[pre])
print(f"East before/after change: {east_change:+.2f} h")
print(f"West change (no playbook): {west_change:+.2f} h")
print(f"Difference-in-differences: {east_change - west_change:+.2f} h")
# robustness: drop month 4 (transition month)
e2 = mean(EAST[4:6]) - mean(EAST[pre]); w2 = mean(WEST[4:6]) - mean(WEST[pre])
print(f"DiD excluding transition month: {e2 - w2:+.2f} h")
# pre-trend check: slope of difference East-West before adoption
diff = [e - w for e, w in zip(EAST, WEST)]
print("East-West gap by month:", [round(d, 2) for d in diff])

Expected output:

East before/after change: -1.90 h
West change (no playbook): -1.30 h
Difference-in-differences: -0.60 h
DiD excluding transition month: -0.55 h
East-West gap by month: [-1.0, -0.9, -0.8, -1.6, -1.4, -1.5]

Interpretation: the naive before/after suggests the playbook cut 1.9 hours. But West, without the playbook, also improved by 1.3 hours (perhaps a new tool or seasonal case mix). The difference-in-differences estimate is about −0.6 hours and similar (−0.55) when the transition month is excluded. The pre-period gap was narrowing slightly (−1.0, −0.9, −0.8), so the parallel-trends assumption is not perfectly met: if the trend had continued, the gap would have narrowed further, which would make the true effect somewhat larger in magnitude than −0.6 under a simple linear extrapolation — another reason to report the estimate with its assumption. Only six monthly medians are used; there is no uncertainty interval here, and case-level analysis would be needed to provide one.

Defensible conclusion: "Compared with a team that did not adopt it, the team using playbook B reduced median containment time by about 0.6 hours more over three months, assuming both teams would otherwise have followed similar trends; the pre-adoption trends were not exactly parallel, and this is not a randomized comparison."

33.8 Common errors and limits

Reporting before/after differences as effects.
Choosing the outcome after seeing results.
Ignoring how adoption was decided.
Presenting point estimates from few aggregated values without uncertainty.
Assuming parallel trends without checking the pre-period, as Lab 33's narrowing gap shows is necessary.
Running robustness checks only until one agrees with the hoped-for result.
Reporting a team-level estimate as if it described each analyst's performance.
Treating six monthly medians as if they carried the precision of hundreds of cases; without case-level data there is no interval.
Ignoring a co-intervention, such as the new tool that may explain part of West's improvement, when it might have affected the two teams differently.

Exercises — chapter 33

Attempt each exercise before opening the worked results. Every case, organization, person, address and number in an exercise is synthetic unless a cited public source is named.

EX097 — Write the estimand. Steps: write the estimand for Lab 33 and two pre-specified robustness checks. Rubric: 3 = population, outcome, contrast, time, assumption; checks pre-specified; 2 = vague population; 1 = outcome only; 0 = none.

EX098 — Pre-trend reasoning. Steps: explain what the narrowing pre-period gap implies for the DiD estimate under a linear trend. Rubric: 3 = gap narrowing by 0.1/month would predict post gaps −0.7, −0.6, −0.5; observed −1.6, −1.4, −1.5 → implied effect ≈ −0.9 h; stated as assumption-dependent; 2 = direction right without numbers; 1 = ignores; 0 = none.

EX099 — Selection. Inputs: East analysts could choose playbook B per case. Steps: explain the bias and propose a design fix. Rubric: 3 = analysts may choose B for simpler cases, inflating its apparent effect; fix by team-level or time-block assignment, or intention-to-treat analysis by team; 2 = bias without fix; 1 = misses; 0 = none.

Chapter synthesis

Fix the estimand before analysis, prefer concurrent comparisons to before/after, record how adoption was assigned, check pre-trends and robustness, and phrase conclusions with their assumptions.