Chapter 29 — Critical infrastructure and sector security: measurement
Lesson map
Lesson 29 published. Map, example, EX057.
flowchart TD
N0["Learning outcomes and connection"]
N1["29.1 What to measure"]
N2["29.2 Detection metrics"]
N3["29.3 Resilience as performance over time"]
N4["29.4 Metrics that mislead"]
N5["29.6 Practical activity: Lab 10"]
N6["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX057"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["29.1 What to measure"]
N2["29.2 Detection metrics"]
N3["29.3 Resilience as performance over time"]
N4["29.4 Metrics that mislead"]
N5["29.6 Practical activity: Lab 10"]
N6["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX057"]
[Learning outcomes and connection]
|
v
[29.1 What to measure]
|
v
[29.2 Detection metrics]
|
v
[29.3 Resilience as performance over time]
|
v
[29.4 Metrics that mislead]
|
v
[29.6 Practical activity: Lab 10]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX057]
Learning outcomes and connection
After this chapter you can (1) define security and resilience metrics that support decisions; (2) compute detection metrics — precision, recall, F1, false-alarm rate, time to detect — from labelled tests; (3) measure resilience as service performance over time through a disruption; and (4) avoid metrics that reward the wrong behaviour. It builds on Chapters 7 and 28 and prepares Chapters 30, 38 and 40.
29.1 What to measure
A metric is useful if it changes a decision. Three families: control performance (does the control do what was specified: rejection rates, detection times); exposure (how much attack surface or vulnerability exists: unauthenticated conduits, unpatched reachable vulnerabilities, shared accounts); and service resilience (how service degrades and recovers). Use the goal–question–metric approach: state the goal, the question that would show progress, and only then the metric.
For TMRP, the alarm latency for R7 measures control performance, the count of unauthenticated conduits measures exposure, and the share of demand met during the recovery trial measures resilience.
29.2 Detection metrics
From a labelled test set: true positives (TP), false positives (FP), false negatives (FN), true negatives (TN). Precision = TP/(TP+FP): of alarms, how many were real. Recall = TP/(TP+FN): of real events, how many were found. F1 is their harmonic mean. Report both precision and recall with the base rate of the test set; precision in a test set with 50% attacks says little about operations with 0.1% (Chapter 7). Time to detect is a distribution; report median and worst cases, not only the mean. Arp et al. (2022) list common pitfalls in evaluating machine-learning security systems, including unrealistic base rates and data snooping; Kim et al. (2022) show how point-adjusted evaluation can inflate time-series anomaly detection scores.
29.3 Resilience as performance over time
Plot service performance $Q(t)$ — for LRWA, the fraction of demand met — through a disruption. Bruneau et al. (2003) proposed measuring resilience loss as the area between full performance and $Q(t)$ from disruption to recovery. The curve shows robustness (how far performance falls), rapidity (how fast it recovers) and the total loss. Engineering requirements map onto it: R4 keeps the drop shallow; R6 shortens the recovery.
TMRP's recovery trial gives a real curve, since supply stayed in local mode while remote operation was restored.
29.4 Metrics that mislead
Counting blocked attacks rewards noisy firewalls; counting patches applied ignores which ones mattered; mean time to detect hides the one event detected after three weeks. When a metric becomes a target, people optimise the metric. Prefer metrics tied to consequences and verified by test.
TMRP's detector acceptance test avoids these by reporting misses and the slowest detection, not just an average. The two missed reconnaissance events stay in the report, because they are the result that changes how much LRWA can rely on the detector.
29.5 Worked example: anomaly detector acceptance test (synthetic)
TMRP's acceptance test of a network anomaly detector uses 14 labelled events from a test week on the bench (Lab 10): 7 malicious (write floods, unauthorised function codes, a new engineering connection) and 7 benign (maintenance, firmware backup, a failover). Results: precision 0.625, recall 0.714, F1 0.667; time to detect for detected events 2–30 minutes, median 7. The two missed attacks were both slow, low-volume reconnaissance. The acceptance decision: deploy for high-volume attack detection, do not rely on it for reconnaissance, and record that the 50% base rate of the test set makes operational precision much lower.
29.6 Practical activity: Lab 10
# Lab 10 - detection metrics and time-to-detect from a labelled synthetic test week (offline)
# (event_id, truly_malicious, alerted, minutes_from_start_to_alert or None)
EV = [(1,1,1,4),(2,1,1,11),(3,1,0,None),(4,0,1,None),(5,0,0,None),(6,1,1,2),(7,0,1,None),
(8,0,0,None),(9,1,1,30),(10,0,0,None),(11,0,0,None),(12,1,0,None),(13,0,1,None),(14,1,1,7)]
tp = sum(1 for e in EV if e[1] and e[2]); fp = sum(1 for e in EV if not e[1] and e[2])
fn = sum(1 for e in EV if e[1] and not e[2]); tn = sum(1 for e in EV if not e[1] and not e[2])
prec, rec = tp / (tp + fp), tp / (tp + fn)
print(f"TP={tp} FP={fp} FN={fn} TN={tn} precision={prec:.3f} recall={rec:.3f} F1={2*prec*rec/(prec+rec):.3f}")
ttd = sorted(e[3] for e in EV if e[1] and e[2])
print("time to detect (min):", ttd, "mean", round(sum(ttd)/len(ttd), 1), "median", ttd[len(ttd)//2])
Expected output:
TP=5 FP=3 FN=2 TN=4 precision=0.625 recall=0.714 F1=0.667
time to detect (min): [2, 4, 7, 11, 30] mean 10.8 median 7
The counts match the worked example: five attacks detected, three benign events alarmed, two attacks missed and four benign events left quiet. The two missed attacks were slow, low-volume reconnaissance, which the detector does not see. The detection times show why a median matters: one event took thirty minutes while most were found in under ten. Because half the test events were malicious, the precision here is far higher than LRWA would see in operation, as EX057 shows.
29.7 Common errors and limits
- Metrics nobody uses for a decision.
- Precision reported from balanced test sets as if operational.
- Mean time to detect without distribution or misses.
- Resilience reported as uptime only.
- Metrics that become targets.
TMRP's acceptance test reports misses and the slowest detection, and its precision is stated with the test set's fifty per cent base rate, so nobody mistakes it for operational precision. Resilience is shown as the share of demand met over time, not as uptime. Every metric in it informs the deployment decision.
Exercises — chapter 29
EX057 — Operational precision. Objective: translate test metrics to operations. Inputs: Lab 10 recall 0.714 and the test FP rate 3/7; operational base rate 1 malicious event per 1,000 events. Steps: compute the operational precision and expected false alarms per 1,000 events; interpret. Expected evidence: numbers and interpretation. Rubric: 3 = correct calculation (precision ≈ 0.0017; ≈ 428 false alarms per 1,000 events), interpretation that the FP estimate from 7 benign events is far too uncertain and the detector unusable without much lower FPR; 2 = numbers right; 1 = wrong formula; 0 = none.
EX058 — Resilience curve. Objective: compute resilience loss. Inputs: synthetic $Q(t)$: 100% until t = 0; 60% from 0 to 4 h; linear recovery to 100% from 4 h to 12 h. Steps: compute the loss area in %·h; recompute if local manual mode (R4) keeps $Q$ at 85% from 0 to 4 h. Expected evidence: two numbers and a sentence. Rubric: 3 = 160+160 = 320 %·h and 60+60 = 120 %·h with clear interpretation; 2 = one error; 1 = method only; 0 = none.
Chapter synthesis
Measure what changes decisions: control performance, exposure and service resilience. Report precision and recall with base rates and time-to-detect distributions including misses. Represent resilience as performance over time. Distrust metrics that can be gamed or that hide the worst case.
Continue studying — chapter 29 of 50
← Chapter 28 · Course chapter index · Chapter 30 →
Continue to the next authored lesson.