Chapter 7 — Probability and statistics
Lesson map
Lesson 07 published. Map, example, EX019.
flowchart TD
N0["Learning outcomes"]
N1["7.1 Rates and intervals"]
N2["7.2 Comparing detectors"]
N3["7.3 Base rates in the control room"]
N4["Practice note"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX019"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["7.1 Rates and intervals"]
N2["7.2 Comparing detectors"]
N3["7.3 Base rates in the control room"]
N4["Practice note"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX019"]
[Learning outcomes]
|
v
[7.1 Rates and intervals]
|
v
[7.2 Comparing detectors]
|
v
[7.3 Base rates in the control room]
|
v
[Practice note]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX019]
Learning outcomes
You will (1) report detection rates with intervals; (2) compare detectors without overclaiming; (3) refuse rankings that the sample cannot support.
7.1 Rates and intervals
Sixty synthetic trials with 0 false alarms and 60/60 detections sounds perfect — until you remember that a Wilson 95% interval for 0/60 still allows a few percent false-alarm rate in the long run. Report the interval, not only the point. For 60/60 detections the interval for the miss rate is also narrow, but it is not zero forever.
Lena's reports for CNHP give the count, the interval and the seed list together, so a reader can see both the result and how much it could move with more trials.
7.2 Comparing detectors
The residual CUSUM lab shows a band detector missing class-A attacks while CUSUM catches them with mean delay ~25 samples, both with 0/60 false alarms on this seed set. That is evidence that CUSUM beats the band on these synthetic attacks, not that CUSUM is universally better. Prefer paired designs on the same traces and state the attack class.
# Lab 27 - residual CUSUM vs band on batch reactor level (synthetic)
import random
def run(seed, atk, det, k=0.8, h=6.0):
rng=random.Random(seed); true=est=50.0; s=0.0; alarm=None
for t in range(500):
inflow=1.0+rng.gauss(0,0.05)+(0.08 if atk and t>=250 else 0)
true+=inflow-0.02*true
lvl=true+rng.gauss(0,0.3); fm=inflow+rng.gauss(0,0.02)
if atk and t>=250: lvl=50+rng.gauss(0,0.3)
if atk=="B" and t>=250: fm=1.0+rng.gauss(0,0.02)
est+=fm-0.02*est
hit=abs(lvl-50)>2.0 if det=="band" else False
if det=="cusum":
s=max(0.0,s+abs(lvl-est)-k); hit=s>h
if hit and alarm is None: alarm=t
return alarm
for det in ("band","cusum"):
fa=sum(run(i,None,det) is not None for i in range(60))
for atk in ("A","B"):
hits=[a-250 for a in (run(i,atk,det) for i in range(60)) if a is not None and a>=250]
print(f"{det} class{atk} FA={fa}/60 det={len(hits)}/60 mean_delay={sum(hits)/len(hits) if hits else float('nan'):.1f}")
Expected output:
band classA FA=0/60 det=0/60 mean_delay=nan
band classB FA=0/60 det=0/60 mean_delay=nan
cusum classA FA=0/60 det=60/60 mean_delay=24.8
cusum classB FA=0/60 det=0/60 mean_delay=nan
Lena's pilot plan keeps the same traces for both detectors and names the attack class tested, so the comparison can be repeated on new seeds later.
7.3 Base rates in the control room
If genuine cyber-physical attacks are rare, a detector with even a 1% false-alarm rate will dominate the operator's attention. Bayes' rule applies exactly as in Chapter 7 of neighbouring security courses: precision collapses when the base rate is tiny. Design for the plant's base rate, not a lab's balanced set.
Synthetic case — Ashford Batch Plant. A vendor claimed "99% detection, 0% false alarms" from a 20-run demo. Lena asked for the attack class, the seed list and the false-alarm interval; none were provided. The pilot used the CUSUM lab design instead.
7.4 Errors and limits
Synthetic attacks are not the adversary. Intervals assume independent trials; correlated seeds invalidate them.
Lena's pilot reports state that the attacks were synthetic, list the seeds and explain how they were generated. If two seeds produce nearly identical traces, she counts them as one trial, because treating them as independent would make the interval look narrower than the evidence supports. Any statistical test she reports is one she actually ran, with its inputs kept so a reviewer can rerun it.
Practice note
Publish the seed list and detector parameters with every detection claim in CNHP documents. Reproducibility is cheaper than arguing about a slide six months later, and it is what Urbina et al. and Giraldo et al. demand of evaluations.
Exercises
- EX019 — Interval. State what 0/60 false alarms lets you claim at ~95%. Rubric: 3 = upper bound ~5–6% (rule of three / Wilson), not "zero forever"; 2 = bound only; 1 = "zero"; 0 = none.
- EX020 — Lab. For CUSUM on class A, report detections and mean delay. Rubric: 3 = 60/60 and ~24.8; 2 = one number; 1 = vague; 0 = wrong.
- EX021 — Ranking refusal. Band and CUSUM both show FA=0/60 on class B (no detections). Write the sentence that refuses calling them equal in quality. Rubric: 3 = cites no power / need class-A or larger n; 2 = vague caution; 1 = declares equal; 0 = none.
Synthesis
Detection claims need intervals, attack class and base-rate honesty — not a single percentage.
Continue studying — chapter 07 of 50
← Chapter 06 (draft) · Course chapter index · Chapter 08 (draft) →