Chapter 2 — Advanced incident response research seminar

Chapter 2 — Advanced incident response research seminar

Lesson map

Lesson 02 published. Map, example, EX008.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["2.1 What the seminar is for"]
  N2["2.2 Competing models of response effectiveness"]
  N3["2.3 How the success measure shapes the answer"]
  N4["2.4 A seven-question reading protocol"]
  N5["2.5 Evidence quality without a hierarchy fetish"]
  N6["2.7 Practical activity: one dataset, three success m"]
  N7["2.8 Writing the disagreement"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX008"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["2.1 What the seminar is for"]
  N2["2.2 Competing models of response effectiveness"]
  N3["2.3 How the success measure shapes the answer"]
  N4["2.4 A seven-question reading protocol"]
  N5["2.5 Evidence quality without a hierarchy fetish"]
  N6["2.7 Practical activity: one dataset, three success m"]
  N7["2.8 Writing the disagreement"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX008"]

[Learning outcomes and connection]
    |
    v
[2.1 What the seminar is for]
    |
    v
[2.2 Competing models of response effectiveness]
    |
    v
[2.3 How the success measure shapes the answer]
    |
    v
[2.4 A seven-question reading protocol]
    |
    v
[2.5 Evidence quality without a hierarchy fetish]
    |
    v
[2.7 Practical activity: one dataset, three success m]
    |
    v
[2.8 Writing the disagreement]
    |
    v
[Worked example] --> [Exercise EX008]

Learning outcomes and connection

After this chapter you should be able to:

1. Describe four competing models of what makes incident response "effective" and the success measure each one implies.
2. Show, with a worked dataset, how the choice of success measure — and the handling of cases that have not finished — can reverse which method looks better.
3. Read a study with a seven-question protocol that separates its design from its result, and grade its evidence for your question.
4. Lead a seminar comparison of two studies and state precisely what each model leaves unexplained.
5. Turn the seminar into a justified choice of primary and secondary outcomes for your own study.

The chapter builds on the concept matrix of Chapter 1 and prepares Chapter 3, where the outcomes chosen here are fixed in a design and a pre-analysis plan.

2.1 What the seminar is for

A master's research seminar is not a reading group that summarizes papers. Its purpose is to make you disagree well: to see that two careful studies of "the same" practice can reach different conclusions because they assumed different things about what response is for, measured different outcomes, or studied different settings. By the end of the seminar you should be able to say which assumptions your own study will adopt and defend that choice to an examiner who prefers another.

The seminar has a rhythm. Before each session, every participant reads the assigned studies using the protocol in §2.4 and writes a one-page note. In the session, one participant presents a study's design (not its conclusion) for ten minutes; a second presents the competing study; the group then works through the questions "what would each study's authors predict about the other's setting?" and "what does each model leave unexplained?". The session ends with a written statement of the disagreement — not a resolution.

2.2 Competing models of response effectiveness

"Effective incident response" has no single definition. Four models recur in the practitioner and research literature. Each is a coherent view; each implies a different success measure; each leaves something out.

The models are not mutually exclusive, and mature programmes use measures from several. NIST SP 800-61 Rev. 3 (April 2025) frames incident response as part of cybersecurity risk management rather than as a self-contained lifecycle, which is closest to the harm-reduction and resilience models, while NIST SP 800-184 (2016) on recovery planning emphasizes restoring and improving capability. Operational SOC dashboards, by contrast, are often built on the speed model, because time stamps are what ticketing systems record automatically. Field studies of SOCs report that metrics chosen for ease of collection shape what analysts optimize (Sundaramurthy et al. 2014; Kokulu et al. 2019).

2.2.1 Decision theories underneath the models

Behind the accuracy model stand two accounts of how analysts decide, introduced in Chapter 1. Situation awareness (Endsley 1995) distinguishes perceiving the relevant elements, comprehending their meaning and projecting their future status. An under-triaged quiet-impact incident, on this account, is a failure of perception or comprehension: the analyst did not attend to the affected service or its time-criticality. A rubric that forces those elements into view should help. Naturalistic decision making (Klein 2008) emphasizes recognition: experienced decision-makers match situations to familiar patterns and simulate a course of action rather than comparing options. On this account, rubrics help when patterns are unfamiliar or misleading but may interfere with expert recognition and add time. A third body of work, on alert overload, argues that the dominant problem is volume and false positives, which teach analysts to discount signals (Alahmadi, Axon and Martinovic 2022).

These accounts make different predictions about a rubric study, which is exactly what makes them useful:

A study that measures only the overall under-triage rate cannot distinguish these accounts. One that also records case type and experience can, at least partly. This is the seminar's practical payoff: the competing models tell you which subgroups and secondary outcomes to predeclare.

2.3 How the success measure shapes the answer

The speed model's measures have properties that are easy to miss and that matter for any study comparing response methods.

Censoring. At the end of a study window some incidents are still open. If they are excluded, the mean time to resolve is computed only on incidents that finished — which are disproportionately the easy ones. A method that leaves more hard cases open will look faster. Survival-analysis methods treat open cases as censored observations (we know they took at least this long); at minimum, report how many were open, and use measures such as the share closed within a fixed time that count every case.

Skew. Resolution times are strongly right-skewed: most incidents close quickly, a few take days. The mean is dominated by the tail; the median describes the typical case; both should be reported, with the share of long cases.

Escalation as cost. A method that correctly escalates more high-impact incidents will generate more long investigations. Under the speed model it looks worse, even though under the accuracy and harm models it is doing its job.

Case mix. If two methods are compared on different incidents, differences in severity mix can dominate any method effect. Chapter 4 returns to this as confounding.

The accuracy model has its own traps. Its reference standard must be set independently of the decisions being judged (otherwise the analyst's rating partly defines the truth); agreement statistics depend on the distribution of categories; and weighted kappa (Cohen 1968) gives partial credit for near misses, which is appropriate for ordinal severity but hides the asymmetry between under- and over-triage. That is why the Triage Rubric Study reports under-triage and over-triage separately, with weighted kappa as a secondary summary.

2.4 A seven-question reading protocol

For every study in your concept matrix, answer these questions in writing before you read its discussion section:

1. Question and model. What did the authors want to know, and which model of effectiveness did they assume?
2. Unit and population. What was a single observation (an incident, a decision, an analyst, a team)? Who or what was sampled, and how?
3. Exposure and assignment. What was compared with what, and who decided which unit got which condition?
4. Outcome and its source. How was the outcome measured, by whom, and could the exposure itself have changed the measurement?
5. Completeness. What happened to units that dropped out, cases still open, or missing fields?
6. Rivals. Which explanations other than the exposure could produce the result, and did the authors test them?
7. Transfer. To which populations and settings does the result plausibly apply, and what is different about yours?

Then grade the study's evidence for your question (not its quality in general): direct (same exposure, comparable population, outcome you will use), indirect (a related exposure, population or outcome), or design-only (useful for methods, not for the answer). The surgical-checklist study of Haynes et al. (2009), for example, is design-only evidence for the Triage Rubric Study: a different profession and outcome, but an instructive pre–post design whose rivals (secular trends, observation effects, simultaneous changes) are the same ones that threaten MRSS's historical data.

2.5 Evidence quality without a hierarchy fetish

Medical research ranks designs in a hierarchy with randomized trials near the top. The ranking captures something real — random assignment breaks the link between exposure and confounders — but applied mechanically it misleads in incident response, where randomized trials on live incidents are almost never feasible or ethical. Better questions are: what is the counterfactual in this study, and how credible is it? and what would the study have shown if the favoured explanation were false? A carefully controlled tabletop with experienced volunteers can provide stronger causal evidence about a decision aid than a large ticket audit, while the ticket audit provides better evidence about how often a problem occurs in practice. Each answers a different question; neither is simply "better".

2.6 Worked example: two studies, two success measures (synthetic)

The seminar is assigned two fictional studies of the same rubric at different organizations.

Study P ("Rubric slows response") compared 300 incidents handled under a structured rubric with 300 handled under usual practice. Its primary outcome was the mean hours to resolve among closed incidents. It reported that rubric incidents took longer (11.3 versus 10.2 hours) and concluded that the rubric "reduces response efficiency".

Study Q ("Rubric reduces missed incidents") analysed the same kind of data with under-triage of high-impact incidents (rating below a panel-reviewed reference) as its outcome. It reported 9.7% versus 23.7% and concluded that the rubric "improves triage".

The seminar's task is not to decide which study is right. It is to show that both can be accurate descriptions of the same data, and to explain why. Lab 2 reconstructs the situation.

2.7 Practical activity: one dataset, three success measures

Lab 2 simulates 300 fictional incidents, each handled once under the rubric (R) and once under usual judgement (U) — a paired simulation, so case mix is identical. Under R, high-impact incidents are under-triaged less often; correctly escalated incidents take longer because escalation triggers containment work; under-triaged high-impact incidents are discovered late, so many are still open when the 72-hour study window closes. Save it as lab2_measures.py.

# Lab 2 - one synthetic dataset, three success measures: which triage method "wins"? (offline, stdlib only)
# The same 300 fictional incidents handled once under each method (a paired simulation). Reference impact: 1 (low) .. 4 (critical), assigned by a review panel.
# Method R (structured rubric) escalates high-impact cases more often; escalated cases take longer to close.
import random, statistics as st
CASES = random.Random(1012).choices([1, 2, 3, 4], weights=[40, 30, 20, 10], k=300)  # same cases, both methods
def simulate(method):
    rng = random.Random({"R": 61, "U": 62}[method])
    rows = []
    for ref in CASES:
        miss_p = {"R": 0.12, "U": 0.30}[method] if ref >= 3 else 0.0   # under-triage probability
        under = rng.random() < miss_p
        escalated = ref >= 3 and not under
        hours = rng.lognormvariate(1.0 + 0.25 * ref, 0.6) + (rng.uniform(10, 26) if escalated else 0.0)
        if under:                                     # missed high-impact cases reopen later, if ever
            hours += rng.lognormvariate(4.3, 0.4)
        closed_by_cutoff = hours <= 72                # study window ends; long cases still open
        rows.append((ref, under, escalated, hours, closed_by_cutoff))
    return rows
data = {m: simulate(m) for m in ("R", "U")}
print("method  closed  mean_h(closed)  median_h(closed)  under-triage(high-impact)  still_open")
for m, rows in data.items():
    closed = [r[3] for r in rows if r[4]]
    hi = [r for r in rows if r[0] >= 3]
    ut = sum(r[1] for r in hi) / len(hi)
    print(f"{m:>6}  {len(closed):>6}  {st.mean(closed):>14.1f}  {st.median(closed):>16.1f}  "
          f"{ut:>17.1%} of {len(hi):>3}  {len(rows) - len(closed):>10}")
for m, rows in data.items():
    allh = sorted(r[3] for r in rows)
    print(f"{m}: median hours including open cases (open = >72h, rank only): {st.median(allh):.1f}"
          f"  share closed within 24h: {sum(h <= 24 for h in allh) / len(allh):.1%}")

Expected output:

method  closed  mean_h(closed)  median_h(closed)  under-triage(high-impact)  still_open
     R     296            11.3               5.2               9.7% of  93           4
     U     280            10.2               5.3              23.7% of  93          20
R: median hours including open cases (open = >72h, rank only): 5.4  share closed within 24h: 82.7%
U: median hours including open cases (open = >72h, rank only): 5.7  share closed within 24h: 80.7%

How to read it. Study P's measure — mean hours among closed incidents — favours U (10.2 versus 11.3 hours). Study Q's measure favours R (under-triage 9.7% versus 23.7% of the same 93 high-impact incidents). Both are arithmetically correct. The reason they disagree is visible in the last column: U leaves 20 incidents open at the cut-off against R's 4, and those open incidents are mostly missed high-impact cases — exactly the ones the speed measure silently drops. The median among closed incidents barely differs, and a measure that counts every incident (share closed within 24 hours) slightly favours R. The lesson is not "Study Q is right": it is that a success measure is a model of what response is for, and a study that does not report what it excluded can make a better method look worse.

2.8 Writing the disagreement

The seminar ends with a written statement in a standard form:

Study P and Study Q agree that the rubric lengthens work on correctly escalated incidents. They disagree about effectiveness because P measures speed among closed incidents and Q measures classification against a reference. P's measure excludes incidents still open at the cut-off, which are disproportionately under-triaged high-impact cases under usual practice; Q's measure depends on the validity of its reference panel and does not show whether better classification reduced harm. Neither study measured harm. Our study will adopt Q's primary outcome, report P's measure with censoring made explicit, and record time per decision as the cost.

Notice what this paragraph does: it names the shared finding, locates the disagreement in the measures, states what each leaves unexplained, and turns that into design decisions.

2.9 Presenting in the seminar

When you present a study, spend most of your time on design: unit, assignment, outcome and rivals. Put the result on one slide at the end. Practise the sentence "the strongest claim this design supports is…". When you challenge a presenter, ask questions that could change the conclusion ("how many incidents were still open?") rather than questions that display your knowledge. Record the session's open questions; several will become your own study's secondary analyses.

Practise that sentence for your own Triage Rubric Study before the seminar, because it is the one examiners will test.

2.10 Common errors and limits

• Treating "effectiveness" as self-evident and adopting whatever metric the ticketing system produces.
• Computing mean time to resolve on closed incidents only, without reporting how many were open or why.
• Reporting means of heavily skewed times without medians or tail shares.
• Comparing methods on different incident sets without addressing case mix.
• Treating agreement with a reference as accuracy without asking how the reference was set and by whom.
• Ranking studies by design label alone instead of asking what counterfactual each provides for your question.
• Ending a seminar with a vote on which study is right instead of a precise statement of the disagreement.

The limit of a seminar is that it works with published reports, which omit much of what you need (excluded cases, measurement details, deviations). Write to authors when a missing detail decides your reading, and record the question even if no reply comes.

Exercises — chapter 2

EX008 — Map the models. Objective: relate success measures to models of effectiveness. Inputs: the table in §2.2. Constraints: one row per model. Steps: for each model, (1) name one measure available from a ticketing system and one that requires extra data collection; (2) give one way the measure could improve while the thing it stands for gets worse. Expected evidence: a table. Rubric: 3 = measures realistic and the "gaming" or distortion examples specific; 2 = one model weak; 1 = measures generic; 0 = none.

EX009 — Extend Lab 2. Objective: see how a study window changes a speed measure. Inputs: Lab 2. Constraints: offline; change only the 72-hour cut-off (both occurrences in the code and the label). Steps: (1) rerun with a 168-hour (one-week) window; (2) report closed counts, mean hours among closed incidents and still-open counts for both methods; (3) explain which direction the mean comparison moved and why; (4) state which reported measures did not change and why. Expected evidence: outputs and a 150-word explanation. Rubric: 3 = outputs correct, the movement explained by late discovery of missed cases entering the closed set, invariance of under-triage explained; 2 = one error; 1 = not run; 0 = none.

EX010 — Predictions from theory. Objective: derive testable predictions from competing decision theories. Inputs: §2.2.1. Constraints: predictions must be observable in a tabletop study. Steps: (1) for each of the three accounts, write one prediction that differs from the other two; (2) say what data the Triage Rubric Study must record to test it; (3) say which prediction the study as designed in Chapter 1 cannot test, and why. Expected evidence: a short table and paragraph. Rubric: 3 = predictions genuinely discriminating, data requirements concrete (case type, experience, time), untestable prediction identified honestly; 2 = predictions overlap; 1 = restates theories; 0 = none.

EX011 — Read a real study with the protocol. Objective: apply the seven-question protocol. Inputs: the published version of Alahmadi, Axon and Martinovic (2022). Constraints: answer from the paper itself; do not rely on summaries. Steps: (1) answer the seven questions; (2) grade its evidence for RQ1 and for RQ3 of the Triage Rubric Study; (3) name one design choice in your study it should influence. Expected evidence: one page. Rubric: 3 = accurate description of a qualitative interview-and-survey study of SOC practitioners, evidence correctly graded as indirect for RQ1 and more direct for RQ3, design influence concrete; 2 = one answer inaccurate; 1 = summary only; 0 = none.

EX012 — The disagreement statement. Objective: write a seminar conclusion. Inputs: Studies P and Q, Lab 2 output. Constraints: 150–200 words; no verdict on which study is "right". Steps: follow the form in §2.8 but add one analysis that, if both studies' authors agreed to it, would narrow the disagreement. Expected evidence: the statement. Rubric: 3 = shared finding, located disagreement, what each leaves unexplained, and a narrowing analysis (for example, time-to-resolve with censored cases handled, or a harm measure); 2 = verdict creeps in; 1 = summary; 0 = none.

EX013 — Choose and justify outcomes. Objective: fix primary and secondary outcomes. Inputs: the four models, Lab 2, your answers to EX010. Constraints: one primary outcome; at most four secondary outcomes. Steps: (1) name and define each outcome exactly; (2) justify the primary in terms of the decision-maker's problem; (3) for each secondary, say what it guards against; (4) name one attractive outcome you will not use and why. Expected evidence: a half-page outcome specification. Rubric: 3 = primary justified by the problem, secondaries guarding against specific misreadings (time cost, over-triage, agreement, subgroup by case type), rejected outcome reasoned; 2 = too many outcomes; 1 = undefined outcomes; 0 = none.

EX014 — Evidence grading. Objective: grade evidence for a specific question. Inputs: the four rows of the concept matrix in §1.6. Constraints: use direct / indirect / design-only. Steps: grade each row for RQ1 and for the historical MRSS question, with one sentence of justification each. Expected evidence: a table. Rubric: 3 = grades differ appropriately between the two questions and the justifications name population, exposure or outcome differences; 2 = one inconsistency; 1 = all graded alike; 0 = none.

Chapter synthesis

What counts as effective incident response depends on what response is for: speed, accuracy, harm reduction or resilience. Each model implies a measure, and each measure leaves something out. Speed measures are distorted by censoring, skew, escalation costs and case mix; accuracy measures depend on an independent reference and should separate under- from over-triage. Competing decision theories predict different patterns across case types and experience, and those predictions tell you which subgroups and secondary outcomes to predeclare. Read studies for design before results, grade their evidence for your question, and end a seminar with a precise statement of disagreement that turns into design decisions.

Continue studying — chapter 02 of 08

← Chapter 01 · Course chapter index · Chapter 03 →

Continue to the next authored lesson.