Chapter 15 — Evidence and measurement
Lesson map
Lesson 15 published. Map, example, EX045.
flowchart TD
N0["Learning outcomes and connection"]
N1["15.1 What a measure is"]
N2["15.2 Chain of transformations"]
N3["15.3 Denominators"]
N4["15.4 Clocks"]
N5["15.5 Missingness"]
N6["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX045"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["15.1 What a measure is"]
N2["15.2 Chain of transformations"]
N3["15.3 Denominators"]
N4["15.4 Clocks"]
N5["15.5 Missingness"]
N6["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX045"]
[Learning outcomes and connection]
|
v
[15.1 What a measure is]
|
v
[15.2 Chain of transformations]
|
v
[15.3 Denominators]
|
v
[15.4 Clocks]
|
v
[15.5 Missingness]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX045]
Learning outcomes and connection
You will be able to (1) trace a measure's provenance through its chain of transformations; (2) identify the denominator a measure actually uses; (3) account for clock differences and missing timestamps; and (4) state what a measure can and cannot support. Chapter 12 gave you statistical tools; here you apply them to the numbers response teams report about themselves. Chapter 29 builds an operational review cadence on these foundations.
15.1 What a measure is
A measure is a defined procedure that turns records into a number: "median minutes from case creation to first analyst action, over cases created in the period". The definition includes the population, the start and end events, the aggregation, and rules for missing values. Changing any of these changes the number. A dashboard label ("time to respond") is not a definition.
The Harbourline claim in the lab shows why: "median time to contain" changes from an improvement to a possible deterioration depending on one rule, how cases without a recorded containment time are treated.
15.2 Chain of transformations
Between raw records and the reported number there are steps: export, filter, join, normalize, compute, round. Each can introduce error or bias. Writing the chain down — ideally as a script — is what makes a measure auditable. This is the same provenance idea as Chapter 8, applied to metrics.
For the Lab 15 claim the chain would read: export cases from the ticketing system, keep those with a containment time, subtract detection time, take the median per period, round to half an hour. Writing it out shows at once that the second step quietly removes six of the twelve cases after the change. A chain kept as a script can be rerun when the source data is corrected, so the number and its method stay together.
15.3 Denominators
Rates and averages depend on which cases are counted. Common hidden denominators: only closed cases (excludes long-running ones), only cases with all timestamps (excludes messy ones, which may be the hard ones), only cases in one queue. Always ask: of how many, out of how many eligible?
Lab 15 shows the effect: the median over recorded cases after the playbook uses six cases, not twelve. A reader told "median fell" deserves to know that half the eligible cases are absent from the calculation and that their absence may not be random.
15.4 Clocks
Different systems have different clocks. Even with time synchronization (the Network Time Protocol is specified in RFC 5905), offsets, time-zone handling and manual entry create discrepancies. A case-management system may record when a ticket was entered, not when the event was detected. Measures that subtract timestamps from different systems inherit both systems' errors.
Exercise EX045 shows how these errors stack: a synchronized SIEM detection time minus a hand-typed containment time in local time can be off by three hours if the zone is mishandled, plus rounding and entry delay. Prefer end events recorded automatically by the system that acted.
15.5 Missingness
Missing values are rarely random. If "time contained" is blank mainly for cases that were never really contained, a mean over non-blank values flatters the team. Report the number missing and test how the conclusion changes under plausible assumptions about them (a simple bound: what if all missing cases took as long as the slowest observed?).
In Lab 15 the missing share rises from two of ten cases to six of twelve after the playbook change, a shift too large to ignore. A plausible cause is a form change that moved the field, but that is a hypothesis to check with the people who fill it in. Recovering missing values from another source beats any bound.
15.6 Worked example and lab (synthetic): "response time improved"
A Harbourline report claims "median time to contain fell from 9.0 to 5.5 hours after the new playbook". Save as lab15.py.
# Lab 15 - audit a response-time claim with missing timestamps (synthetic, stdlib only)
from statistics import median
# hours from detection to containment; None = containment time not recorded
BEFORE = [3, 5, 7, 9, 9, 11, 14, 20, None, None]
AFTER = [2, 3, 5, 6, 7, 8, None, None, None, None, None, None]
def audit(name, xs, worst):
obs = [x for x in xs if x is not None]
miss = len(xs) - len(obs)
filled = obs + [worst] * miss
print(f"{name}: n={len(xs)} recorded={len(obs)} missing={miss} "
f"median(recorded)={median(obs)} median(missing=worst {worst}h)={median(filled)}")
audit("before", BEFORE, 24)
audit("after ", AFTER, 24)
Expected output:
before: n=10 recorded=8 missing=2 median(recorded)=9.0 median(missing=worst 24h)=10.0
after : n=12 recorded=6 missing=6 median(recorded)=5.5 median(missing=worst 24h)=16.0
Interpretation: the claim uses only recorded cases. After the playbook, half the cases have no containment time — perhaps because the new playbook's form moved the field. If the missing cases took as long as a 24-hour bound, the median after would be worse (16.0 h) than before (10.0 h). The data therefore cannot support "improved". What is needed: why the values are missing (form change? cases never contained?), and recovery of the missing times from other sources (change tickets, chat logs) before any claim. The defensible statement: "Among cases with recorded containment times, the median fell from 9.0 to 5.5 hours; containment time is missing for 6 of 12 cases after the change versus 2 of 10 before, so no conclusion about improvement is possible yet."
15.7 Common errors and limits
- Reporting a median over recorded values without the count missing.
- Comparing periods whose recording practice changed.
- Subtracting timestamps from systems with unknown clock offsets.
-
Bounds are not estimates: the 24-hour fill shows how far the answer could move, not what happened.
-
Reading the missing-value rate as a data-entry nuisance rather than a possible signal about the process being measured.
- Choosing a bound after seeing which value supports the preferred conclusion.
- Reporting a single figure such as five and a half hours without the period, population and definition that produced it.
Exercises — chapter 15
Attempt each exercise before opening the worked results. Every case, organization, person, address and number in an exercise is synthetic unless a cited public source is named.
EX043 — Run and bound. Steps: run Lab 15 with a gentler bound (missing = 12 hours) and report whether the conclusion changes. Rubric: 3 = before 10.0, after 10.0 with the arithmetic shown, conclusion still "cannot claim improvement"; 2 = numbers right, conclusion weak; 1 = not run; 0 = none.
EX044 — Write the definition. Steps: write a complete definition of "time to contain" for Harbourline, including start event, end event, source systems, population and missing-value rule. Rubric: 3 = all five elements, clock source named, exclusion rules explicit; 2 = one missing; 1 = label only; 0 = none.
EX045 — Clock offsets. Inputs: detection time from the SIEM (UTC, NTP-synchronized); containment time typed manually into the ticket in local time (UTC−3) to the nearest 15 minutes. Steps: state the error sources and their likely size. Rubric: 3 = time-zone conversion risk (3 h if wrong), rounding (up to ±7.5 min), entry delay (unknown, possibly hours), SIEM ingestion delay; proposes recording containment from the system that performed it; 2 = two sources; 1 = one; 0 = none.
Chapter synthesis
A number is a procedure. Define it, script its transformations, show its denominator, reconcile its clocks, count its missing values and bound how far they could move the conclusion.
Continue studying — chapter 15 of 40
← Chapter 14 · Course chapter index · Chapter 16 →
Continue to the next authored lesson.