Chapter 9 — Evidence and measurement
Lesson map
Lesson 09 published. Map, example, EX025.
flowchart TD
N0["Learning outcomes and connection"]
N1["9.1 What a good measure states"]
N2["9.2 Distributions, not just averages"]
N3["9.3 Open items and survivorship"]
N4["9.4 Small samples need intervals"]
N5["9.6 Practical activity: honest measures"]
N6["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX025"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["9.1 What a good measure states"]
N2["9.2 Distributions, not just averages"]
N3["9.3 Open items and survivorship"]
N4["9.4 Small samples need intervals"]
N5["9.6 Practical activity: honest measures"]
N6["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX025"]
[Learning outcomes and connection]
|
v
[9.1 What a good measure states]
|
v
[9.2 Distributions, not just averages]
|
v
[9.3 Open items and survivorship]
|
v
[9.4 Small samples need intervals]
|
v
[9.6 Practical activity: honest measures]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX025]
Learning outcomes and connection
After this chapter you should be able to:
- choose a measure that answers an operational question, and state its numerator, denominator and period;
- report distributions with medians and high percentiles, and say what an average hides;
- include what has not happened yet (open items) in latency measures;
- report a success rate from a small number of trials with an honest interval.
Every later chapter asks you to prove something with evidence: that a control works, that patching is timely, that a restore meets its target. Measurement is how an implementer turns "it's fine" into a claim someone can check. It is also how organizations fool themselves — usually not by lying, but by choosing a convenient denominator.
9.1 What a good measure states
A measure is useful when someone else could recompute it. State the question ("how quickly do we install vendor fixes on station controllers?"), the population (which assets, which fixes), the numerator and denominator, the period, the source of the data, and what is excluded and why. A patch-compliance figure of "96 %" with no denominator might mean 96 % of office laptops, which says nothing about the pumping stations.
Lab 9's pooled 9-day median fails this test: it gives no population, and its denominator silently excludes seven open fixes.
9.2 Distributions, not just averages
Operational quantities — time to patch, time to restore, alert response time — are usually skewed: most are quick, a few are very slow, and the slow ones are the ones that matter. The median describes the typical case; the 90th percentile (p90) describes the bad cases; the mean is pulled by outliers and describes neither well. Report median and p90 together, and look at the maximum.
In Lab 9, the station controllers' median of 95 days with five fixes still open is the figure the pooled median hid.
9.3 Open items and survivorship
A common error in latency measures is to compute them only over completed items. If five station-controller fixes have been released but not installed, they are the slowest of all — and they are invisible to a median computed over installed fixes. Always report how many items are still open and how old they are. NIST's guidance on enterprise patch management treats patching as a risk-response process with measurable timeliness and recommends metrics that reflect it (NIST SP 800-40 Rev. 4).
Pooling groups with very different behaviour is the related trap. A single median across office IT, servers and station controllers is dominated by whichever group has the most items — usually office IT — and hides the group with the highest consequence.
9.4 Small samples need intervals
Critical-service tests are expensive, so samples are small: ten restore drills, eight injected test alerts. "9 out of 10 drills met the target" is a fact; "our restore success rate is 90 %" is an over-claim. A confidence interval shows the range of underlying rates consistent with the data. For proportions from small samples, the Wilson score interval behaves well (Wilson, 1927). For 9 of 10, the 95 % Wilson interval is roughly 0.60 to 0.98: the true rate could plausibly be as low as 60 %.
This is why Lab 23's backup restore, 9 of 10 drills on target, still fails a claim rule whose lower bound must reach 0.70.
9.5 Worked example: patch latency and restore drills (synthetic)
FSTS's monthly report says "median patch latency: 9 days". Lab 9 recomputes it by group. Corporate IT: 12 of 12 fixes installed, median 5.5 days, p90 about 9. Relay/historian: 6 of 8 installed, median 18.5 days, p90 about 31.5, two still open. Station PLC/RTU: only 3 of 8 installed, median 95 days, five still open. The pooled median of 9 days is arithmetically correct and operationally misleading: it describes laptops, and it omits the seven open fixes — five of them on the highest-consequence assets.
Station controllers are slower to patch for good reasons: vendors must certify fixes, installation may need a site visit and an outage window agreed with operations (Chapter 16). The measure should not punish that; it should show it, so that compensating controls can be justified for the period of exposure.
9.6 Practical activity: honest measures
Lab 9 (save as lab09.py, run with python3 lab09.py):
# Lab 9 - measuring patch latency and a restore drill honestly (offline, standard library only)
# Synthetic: days from vendor fix release to installation, Fallowmere estate, last two quarters.
import math, statistics as st
latency = {"corp IT": [3, 5, 4, 9, 6, 2, 7, 5, 30, 4, 6, 8],
"relay/historian": [12, 19, 15, 41, 22, 18],
"station PLC/RTU": [64, 140, 95]}
not_yet = {"corp IT": 0, "relay/historian": 2, "station PLC/RTU": 5} # fixes released but not installed
def pct(xs, p):
xs = sorted(xs); k = (len(xs) - 1) * p; f = math.floor(k); c = math.ceil(k)
return xs[f] + (xs[c] - xs[f]) * (k - f)
for g, xs in latency.items():
n = len(xs) + not_yet[g]
print(f"{g:<16} installed {len(xs):>2}/{n:<2} median {st.median(xs):>5.1f} d p90 {pct(xs, .9):>6.1f} d mean {st.mean(xs):>5.1f} d")
allx = [x for xs in latency.values() for x in xs]
print(f"pooled (misleading) median {st.median(allx):.1f} d over {len(allx)} installed fixes; {sum(not_yet.values())} still open are invisible to it")
def wilson(k, n, z=1.96):
p = k / n; d = 1 + z*z/n; c = (p + z*z/(2*n)) / d; h = z*math.sqrt(p*(1-p)/n + z*z/(4*n*n)) / d
return c - h, c + h
k, n = 9, 10 # restore drills that met the 4-hour target
lo, hi = wilson(k, n)
print(f"restore drills within target: {k}/{n} = {k/n:.0%}, 95% Wilson interval [{lo:.2f}, {hi:.2f}]")
Expected output (Python 3.13.5):
corp IT installed 12/12 median 5.5 d p90 8.9 d mean 7.4 d
relay/historian installed 6/8 median 18.5 d p90 31.5 d mean 21.2 d
station PLC/RTU installed 3/8 median 95.0 d p90 131.0 d mean 99.7 d
pooled (misleading) median 9.0 d over 21 installed fixes; 7 still open are invisible to it
restore drills within target: 9/10 = 90%, 95% Wilson interval [0.60, 0.98]
Write the report line you would send instead of "median patch latency: 9 days". It should fit in two sentences.
A two-sentence line that would pass: "Corporate IT installed 12 of 12 fixes, median 5.5 days; relay and historian 6 of 8, median 18.5 days, two open. Station controllers installed 3 of 8, median 95 days, with five fixes open under compensating controls and dated plans." It is longer than "9 days", but a reader can now see where the risk sits, and the station controllers are no longer hidden behind office laptops.
9.7 Common errors and limits
- Reporting a percentage without its denominator.
- Averages for skewed quantities.
- Computing latency only over closed items.
- Pooling groups so that the large, low-consequence group hides the small, high-consequence one.
- Turning a small number of successful trials into a rate without an interval.
- Changing a measure's definition between reports without saying so.
Measures describe what was recorded. If the source data are incomplete — fixes not recorded, drills not logged — no statistic will repair that.
In Lab 9 the pooled median shows three of these at once: no denominator, open items excluded and groups pooled.
Exercises — chapter 9
EX025 — Rewrite the report line. Objective: communicate a measure honestly. Inputs: Lab 9 output. Constraints: at most two sentences; include denominators and open items. Steps: write it, then list what you left out and why. Expected evidence: the line and a note. Rubric: 3 = by-group medians or p90 with installed/total counts, open station fixes explicitly mentioned, no pooled figure presented as the headline; 2 = mostly; 1 = repeats the pooled median; 0 = none.
EX026 — Age the open items. Objective: include open items in the measure. Inputs: the five open station fixes were released 20, 45, 70, 130 and 200 days ago; the two open relay/historian fixes 10 and 35 days ago. Constraints: treat each open fix's age as a lower bound on its latency. Steps: (1) combine installed latencies with open ages for station PLC/RTU; (2) compute a median that is a lower bound; (3) explain why this is a lower bound and not an estimate. Expected evidence: calculation and explanation. Rubric: 3 = combined values 20, 45, 64, 70, 95, 130, 140, 200 → median lower bound 82.5 days, correctly explained (open items will only get older); 2 = arithmetic slip; 1 = ignores open items; 0 = none.
EX027 — How many drills? Objective: plan sample size from an interval. Inputs: the Wilson function in Lab 9. Constraints: assume every drill succeeds. Steps: (1) find the smallest number of consecutive successful drills for which the 95 % Wilson lower bound is at least 0.80; (2) say whether that is affordable at one drill per month and what you would do instead. Expected evidence: the number with the run that shows it and a paragraph. Rubric: 3 = 16 drills (lower bound ≈ 0.806; 15 gives ≈ 0.796), recognized as 16 months at one per month, with sensible alternatives such as smaller, more frequent partial restores or automated restore tests; 2 = right number, no plan; 1 = guessed; 0 = none.
Chapter synthesis
A measure is useful when it states its question, population, numerator, denominator, period, source and exclusions. Report medians with high percentiles for skewed quantities; include open items; do not pool groups with different consequences; and attach intervals to small-sample success rates. The aim is not a flattering number but one that someone else can recompute and act on.
Continue studying — chapter 09 of 30
← Chapter 08 · Course chapter index · Chapter 10 →
Continue to the next authored lesson.