Chapter 9 — Evidence and measurement | Technologist Incident response and cyber resilience

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 09 published. Map, example, EX025.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["9.1 Two kinds of evidence"]
  N2["9.2 Defining a measure"]
  N3["9.3 Honest reporting"]
  N4["9.4 Lab T-09 synthetic : triage latency"]
  N5["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX025"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["9.1 Two kinds of evidence"]
  N2["9.2 Defining a measure"]
  N3["9.3 Honest reporting"]
  N4["9.4 Lab T-09 synthetic : triage latency"]
  N5["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX025"]

[Learning outcomes and connection]
    |
    v
[9.1 Two kinds of evidence]
    |
    v
[9.2 Defining a measure]
    |
    v
[9.3 Honest reporting]
    |
    v
[9.4 Lab T-09 synthetic : triage latency]
    |
    v
[Chapter synthesis]
    |
    v
[Worked example] --> [Exercise EX025]

Learning outcomes and connection

You will be able to (1) distinguish evidence preserved for a case from data measured about the service; (2) define a small set of response-service measures with exact definitions; and (3) compute and interpret latency measures honestly, including missing data. This closes the first-year foundations. From Chapter 10 onward you will build the service, and you will measure it with the definitions set here.

9.1 Two kinds of evidence

Case evidence answers "what happened in this incident?" — logs, images, artefacts, preserved with hashes and custody (Chapter 17). Service evidence answers "how well does our response service work?" — times, counts, completeness of records, outcomes of exercises. The first serves investigation and possibly legal processes; the second serves management and improvement. Both need provenance. For example, the hashed log exports in the permits-clerk-4 case are case evidence, while the time from alert to triage in that case is one data point of service evidence. The two are stored differently and serve different readers.

9.2 Defining a measure

A useful measure has: a name; a precise start and end event (for example, "time to triage = triage-start time minus alert time, both from case-system fields"); a population (which cases); handling of missing values; a reporting statistic (median and a high percentile rather than mean, because response times are skewed); a target with rationale; and an owner. Without exact definitions, two teams will report different numbers for the "same" metric. Fernmoor's charter targets show why: acknowledging S1 within 15 minutes can be measured only if the case system records both the page time and the acknowledgment time.

9.3 Honest reporting

Report how many cases were measured and how many were missing. Separate out-of-hours from in-hours if the service model differs. Small numbers vary a lot; seven cases do not establish a trend. Avoid incentives that improve the number but not the service — for example, clicking "triage start" before actually looking. Fernmoor's out-of-hours figures illustrate the point. A small number of night cases can push a percentile above target in one month and below it in the next, so the service report shows the count of cases beside each figure and avoids drawing a trend from a single month. Missing triage times are listed as a separate count, not dropped.

9.4 Lab T-09 (synthetic): triage latency

The script below computes time to triage for eight synthetic Fernmoor cases, using RFC 3339 timestamps. Predict the median before running it.

# Lab T-09: triage latency from synthetic Fernmoor case records (stdlib only)
from datetime import datetime
from statistics import median
cases = [  # case id, alert time, triage start (UTC, RFC 3339)
 ("FCS-101","2026-03-02T08:00:00Z","2026-03-02T08:12:00Z"),
 ("FCS-102","2026-03-02T09:30:00Z","2026-03-02T09:36:00Z"),
 ("FCS-103","2026-03-02T22:10:00Z","2026-03-03T00:05:00Z"),
 ("FCS-104","2026-03-03T07:45:00Z","2026-03-03T07:59:00Z"),
 ("FCS-105","2026-03-03T13:20:00Z",None),
 ("FCS-106","2026-03-04T02:00:00Z","2026-03-04T03:10:00Z"),
 ("FCS-107","2026-03-04T10:00:00Z","2026-03-04T10:09:00Z"),
 ("FCS-108","2026-03-04T11:00:00Z","2026-03-04T11:21:00Z"),
]
p = lambda s: datetime.fromisoformat(s.replace("Z","+00:00"))
mins = {c:(p(t)-p(a)).total_seconds()/60 for c,a,t in cases if t}
missing = [c for c,a,t in cases if not t]
v = sorted(mins.values())
print("measured", len(v), "of", len(cases), "missing", missing)
print("median minutes", median(v))
k = max(1, round(0.9*len(v)))  # nearest-rank p90
print("p90 minutes (nearest rank)", v[k-1])
print("within 15 min target", sum(x<=15 for x in v), "of", len(v))
off = [c for c,a,t in cases if t and not 7 <= p(a).hour < 19]
print("out-of-hours alerts", off, "minutes", [mins[c] for c in off])

Expected output:

measured 7 of 8 missing ['FCS-105']
median minutes 14.0
p90 minutes (nearest rank) 70.0
within 15 min target 4 of 7
out-of-hours alerts ['FCS-103', 'FCS-106'] minutes [115.0, 70.0]

Interpretation: the median (14 minutes) meets a 15-minute target, but only 4 of 7 measured cases did, the nearest-rank 90th percentile is 70 minutes, and both slow cases were out of hours. One case had no triage time at all — itself a finding about record completeness. The honest summary: "In-hours triage is usually within target; out-of-hours triage is not; one record is incomplete; numbers are too few for a trend." That points to a specific improvement (on-call paging, Chapter 21) rather than a general exhortation.

9.5 Common errors and limits

Reporting means of skewed times.
Hiding missing records instead of counting them.
Changing a definition mid-year without restating earlier figures.
Using service measures to blame individuals, which corrupts the data.

Each of these misleads a different reader. A mean triage time can look acceptable while a few out-of-hours cases wait far too long; hidden missing records make completeness look perfect; a silent change of definition makes this year incomparable with last; and using the figures against named analysts makes them record times defensively. Fernmoor's report avoids all four by design.

Exercises — chapter 9

Attempt each exercise before opening the worked results. Every case, organization, person, address and number in an exercise is synthetic unless a cited public source is named.

EX025 — Definition card. Steps: write a definition card for "time to contain". Rubric: 3 = start/end events and fields, population, missing-value rule, statistics, target and owner; 2 = missing two elements; 1 = name only; 0 = none.

EX026 — Re-run with a change. Steps: in the lab, give FCS-105 a triage start of 13:31 and recompute median, p90 and within-target count. Rubric: 3 = 8 measured, median 13.0, p90 70.0, 5 of 8 within 15; 2 = one value wrong; 1 = attempted; 0 = none.

EX027 — Summary for the manager. Steps: write a four-sentence summary of the original lab output. Rubric: 3 = median and percentile, within-target count, out-of-hours finding, missing record, caution on small n; 2 = omits one; 1 = mean or single number; 0 = none.

Chapter synthesis

Keep case evidence and service evidence distinct; define each measure exactly; report medians, percentiles, counts and missing values; separate structural conditions such as out-of-hours; and turn the result into a specific, testable improvement.