Chapter 29 — OT, IoT and cyber-physical security: measurement
Lesson map
Lesson 29 published. Map, example, EX085.
flowchart TD
N0["Learning outcomes"]
N1["29.1 Coverage needs a plan"]
N2["29.2 Activity vs outcome"]
N3["29.3 Detector honesty"]
N4["Practice note"]
N5["29.1b Sampling and cohort comparisons"]
N6["Field note — methodology lines on charts"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX085"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["29.1 Coverage needs a plan"]
N2["29.2 Activity vs outcome"]
N3["29.3 Detector honesty"]
N4["Practice note"]
N5["29.1b Sampling and cohort comparisons"]
N6["Field note — methodology lines on charts"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX085"]
[Learning outcomes]
|
v
[29.1 Coverage needs a plan]
|
v
[29.2 Activity vs outcome]
|
v
[29.3 Detector honesty]
|
v
[Practice note]
|
v
[29.1b Sampling and cohort comparisons]
|
v
[Field note — methodology lines on charts]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX085]
Learning outcomes
You will (1) define coverage with a denominator; (2) separate activity metrics from outcome metrics; (3) evaluate detectors with attack class and intervals (Urbina/Giraldo lessons).
29.1 Coverage needs a plan
ATT&CK for ICS coverage is observed intersect planned over planned, not observed over all techniques. Without a plan, coverage is vanity.
At ABP, Lena writes the plan of techniques to be tested before each assessment, and the coverage board counts only those. Observed techniques outside the plan are reported separately. The figure of 31 of 44 planned techniques observed means something because the 44 were chosen in advance and the planning guide is published with the board. The plan is fixed before testing.
29.2 Activity vs outcome
Activity: frames decoded, tools launched. Outcome: verified remediations, SIS reachability still false, false-alarm rate at plant base rate. Steer on outcomes.
Lena's monthly CNHP report leads with outcomes: remediations verified by re-test, whether SIS-B is still unreachable from L4, and the detector's false-alarm rate at ABP's base rate. Activity counts, such as frames decoded, appear only in a capacity appendix. Steering on activity would reward running more tools rather than fixing more problems. Outcomes are what the plant manager funds.
29.3 Detector honesty
Report false-alarm intervals, detection delay, and attack class. A band detector that misses stealthy residual attacks is not equivalent to a CUSUM that catches them — the lab makes that visible.
# Lab 27 - residual CUSUM vs band on batch reactor level (synthetic)
import random
def run(seed, atk, det, k=0.8, h=6.0):
rng=random.Random(seed); true=est=50.0; s=0.0; alarm=None
for t in range(500):
inflow=1.0+rng.gauss(0,0.05)+(0.08 if atk and t>=250 else 0)
true+=inflow-0.02*true
lvl=true+rng.gauss(0,0.3); fm=inflow+rng.gauss(0,0.02)
if atk and t>=250: lvl=50+rng.gauss(0,0.3)
if atk=="B" and t>=250: fm=1.0+rng.gauss(0,0.02)
est+=fm-0.02*est
hit=abs(lvl-50)>2.0 if det=="band" else False
if det=="cusum":
s=max(0.0,s+abs(lvl-est)-k); hit=s>h
if hit and alarm is None: alarm=t
return alarm
for det in ("band","cusum"):
fa=sum(run(i,None,det) is not None for i in range(60))
for atk in ("A","B"):
hits=[a-250 for a in (run(i,atk,det) for i in range(60)) if a is not None and a>=250]
print(f"{det} class{atk} FA={fa}/60 det={len(hits)}/60 mean_delay={sum(hits)/len(hits) if hits else float('nan'):.1f}")
Expected output:
band classA FA=0/60 det=0/60 mean_delay=nan
band classB FA=0/60 det=0/60 mean_delay=nan
cusum classA FA=0/60 det=60/60 mean_delay=24.8
cusum classB FA=0/60 det=0/60 mean_delay=nan
Synthetic case — Ashford Batch Plant. CNHP reports planned-technique coverage and detector FA/delay by class — not a single secure percent.
Lena's pilot report for ABP lists results by attack class. Each class has its own false-alarm interval and mean delay, so a reader can see that CUSUM caught the stealthy residual attacks the band detector missed.
29.4 Errors and limits
Denominators stolen from another plant invalidate the metric. Synthetic attacks are not the adversary.
Lena never borrows a denominator from another site's plan, because ABP's techniques, assets and processes differ. If a comparison with another plant is needed, both sites must use the same planning rules, and the report says so. She also states in each report that synthetic attacks represent chosen classes, not the behaviour of a real adversary, so readers do not overrate the detection figures. Both caveats appear in every report.
Practice note
Caption every dashboard widget with the formula and the plant id. A widget on the CNHP dashboard without a formula is decoration and is removed in review.
29.1b Sampling and cohort comparisons
If you sample 50 of 500 assets, say so and do not scale findings as if you visited all 500. Comparing coverage across windows requires the same planning rules. Publish the planning guide beside the coverage board so 71% in March means the same kind of denominator as 71% in September.
Sample selection is recorded with its seed or rule, so the same assets can be revisited at re-test. Prefer metrics that under-claim success, such as counting a fix only after re-test.
ABP's planning guide is published beside the board.
Field note — methodology lines on charts
When CNHP changed planning rules in June, Pulse-style charts gained a vertical line labelled methodology_id=2026-06. Leadership stopped comparing May's 80% to July's 70% as if they meant the same denominator. Honesty in the chart legend prevented a false narrative of decline.
Exercises
- EX085 — Formula. Write coverage formula and explain a bad denominator. Rubric: 3 = planned denominator + example of all-ICS misuse; 2 = formula; 1 = vanity; 0 = none.
- EX086 — Lab. Compare band vs CUSUM on class A in one sentence. Rubric: 3 = CUSUM detects, band does not, FA both 0 in sample; 2 = partial; 1 = wrong; 0 = none.
- EX087 — Under-claim. One design that under-claims success. Rubric: 3 = require re-test before closed; 2 = partial; 1 = vague; 0 = none.
Synthesis
Plan the denominator; measure outcomes; evaluate detectors by class with intervals.
Continue studying — chapter 29 of 50
← Chapter 28 (draft) · Course chapter index · Chapter 30 (draft) →