Chapter 29 — Measure triage accuracy and response latency | Engineering Incident response and cyber resilience

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 29 published. Map, example, EX057.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["29.1 Define before measuring"]
  N2["29.2 Accuracy measures"]
  N3["29.3 Latency distributions"]
  N4["29.4 Before-and-after designs and confounding"]
  N5["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX057"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["29.1 Define before measuring"]
  N2["29.2 Accuracy measures"]
  N3["29.3 Latency distributions"]
  N4["29.4 Before-and-after designs and confounding"]
  N5["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX057"]

[Learning outcomes and connection]
    |
    v
[29.1 Define before measuring]
    |
    v
[29.2 Accuracy measures]
    |
    v
[29.3 Latency distributions]
    |
    v
[29.4 Before-and-after designs and confounding]
    |
    v
[Chapter synthesis]
    |
    v
[Worked example] --> [Exercise EX057]

Learning outcomes and connection

After this chapter you can (1) define the population and unit of analysis for evaluating a triage workflow; (2) compute precision, recall and latency distributions with uncertainty; (3) handle missing data and case-mix differences honestly; and (4) perform a sensitivity check that tests whether the conclusion survives reasonable alternative choices. It builds on Chapters 7, 21 and 24 and prepares Chapters 41, 42 and 49.

29.1 Define before measuring

An evaluation states: the population (which alerts or cases, which period, which tenants); the unit (alert, case or entity — results differ); the reference standard (how "true incident" is decided, by whom, blind to which workflow handled it); the primary measures and their thresholds for a meaningful difference; and how missing outcomes are treated. Write these before looking at results. The HVSS evaluation below names its population as the 1,200 alerts of six weeks and its reference standard as a review panel blind to workflow.

29.2 Accuracy measures

For escalation decisions against the reference standard: precision = escalated-and-true / escalated; recall = escalated-and-true / all true; report both with intervals (Chapter 7) and the prevalence in the evaluated population. Arp et al. (2022) warn that unrealistic base rates in evaluation sets inflate apparent performance.

In the HVSS trial below, prevalence is 96 true incidents in 1,200 alerts, or 8%. That is far above the 0.5% assumed in the Lab 7 PPV line, so its precision figures would not carry over to a lower-prevalence source without recomputing them.

29.3 Latency distributions

Report latency as a distribution: median, 90th and 99th percentiles, and the count of cases never actioned within the observation window (censored). A mean hides the long tail where harm occurs. If 5% of cases never close within the window, compute percentiles including them as "greater than window", not by dropping them.

The HVSS trial shows why the tail matters. v2 shortened the median escalation latency by only four minutes, from 22 to 18. At the 90th percentile it cut 95 minutes to 61, which is where slow decisions on real incidents do their damage.

29.4 Before-and-after designs and confounding

Comparing workflow v1 (March) with v2 (May) confounds the workflow with everything else that changed: case mix, staffing, threat activity. Better designs: run both concurrently with random assignment; or alternate by week; at minimum, stratify by severity and source and report standardised results (Chapter 7). Causal reasoning with explicit assumptions is treated in Hernán and Robins, Causal Inference: What If.

This is why the HVSS v1 and v2 trial ran both workflows concurrently for six weeks with seeded random assignment after severity stratification. A March-versus-May comparison would have mixed the workflow change with any shift in staffing or threat activity between those months.

29.5 Worked example (synthetic): HVSS triage v1 vs v2

HVSS ran v1 and v2 concurrently for six weeks, assigning incoming alerts at random (seeded) after severity stratification. Of 1,200 alerts, a review panel blind to workflow labelled 96 as true incidents. Results: v1 precision 0.31 (CI 0.24–0.39), recall 0.79; v2 precision 0.44 (CI 0.35–0.53), recall 0.81. Latency to escalation decision: v1 median 22 min, p90 95 min; v2 median 18 min, p90 61 min. Thirty alerts (2.5%) lacked a panel label because raw evidence had expired before review — missing data.

Sensitivity check. Treat all 30 unlabelled as (a) false, (b) true, (c) distributed like labelled ones. Under (b), v2's precision advantage narrows from 0.13 to 0.09 but remains, because unlabelled alerts were evenly split between workflows; the recall difference reverses sign under one extreme. Conclusion: v2 improves precision and tail latency; no claim about recall. The missing labels also produced an engineering finding — evidence retention must exceed the review cycle.

29.6 Errors and limits

Changing the definition of "true" after seeing results.
Means for latency; dropping censored cases.
Before-after comparisons presented as causal.
Ignoring missing outcomes.

The HVSS trial avoided each error by design, which is what makes its limited conclusion credible. The panel's labels were fixed blind to workflow, so the meaning of "true" could not move after results were seen. Latency was reported as median and p90 rather than as a mean. The 30 unlabelled alerts were tested under three assumptions instead of being dropped, and that test is why the report makes no claim about recall.

Exercises — chapter 29

EX057 — Censored latency. Objective: compute percentiles honestly. Inputs: 20 synthetic latencies in minutes: 5, 7, 8, 9, 10, 12, 12, 14, 15, 18, 20, 22, 25, 30, 41, 55, 70, 90, and two not actioned within 120 min. Steps: compute median and p90 treating unactioned as >120; compare with dropping them. Expected evidence: numbers. Rubric: 3 = with the two unactioned cases included as >120: median 19 (mean of the 10th and 11th values, 18 and 20) and nearest-rank p90 = 90 (the 18th of 20, with the 19th and 20th censored); dropping them gives median 16.5 and p90 ≈ 70–72 — an optimistic picture, with interpretation; 2 = partial; 1 = method; 0 = none.

EX058 — Evaluation protocol. Objective: pre-specify. Inputs: a planned v3 workflow. Steps: write a one-page protocol: population, unit, assignment, reference standard, primary measures with minimum meaningful difference, missing-data rule, sensitivity checks. Expected evidence: protocol. Rubric: 3 = complete and pre-specified; 2 = one gap; 1 = vague; 0 = none.

Chapter synthesis

Evaluate response workflows with a pre-specified population, unit, reference standard and measures; report precision and recall with intervals and prevalence, latency as distributions including censored cases, and test whether conclusions survive missing-data and design alternatives.