Chapter 6 — OT, IoT and cyber-physical security: controlled evaluation
Lesson map
Lesson 06 published. Map, example, EX026.
flowchart TD
N0["Learning outcomes"]
N1["6.1 Same data, two treatments"]
N2["6.2 Reading the result"]
N3["6.3 Making the call"]
N4["6.4 Threats to validity"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX026"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["6.1 Same data, two treatments"]
N2["6.2 Reading the result"]
N3["6.3 Making the call"]
N4["6.4 Threats to validity"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX026"]
[Learning outcomes]
|
v
[6.1 Same data, two treatments]
|
v
[6.2 Reading the result]
|
v
[6.3 Making the call]
|
v
[6.4 Threats to validity]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX026]
Learning outcomes
You will (1) design a controlled comparison of two alerting rule sets on the same labelled replay; (2) estimate recall and false-alert rate for each; (3) compute a bootstrap confidence interval for the difference; and (4) decide whether the evidence justifies a change in production.
6.1 Same data, two treatments
Comparing rule set A (current) with rule set B (proposed) on different weeks of data confounds the rules with the weeks. A controlled evaluation replays the same labelled windows through both. Labels come from a test plan: 40 windows contain injected attack behaviour in the replay, 160 are normal operation.
"""Lab 06: controlled evaluation of two alerting rule sets on the same synthetic replay, with bootstrap CI."""
import random
rnd = random.Random(6)
# 200 labelled replay windows: (is_attack, alert_by_A, alert_by_B)
data = []
for i in range(200):
att = i < 40
a = rnd.random() < (0.70 if att else 0.08)
b = rnd.random() < (0.85 if att else 0.05)
data.append((att, a, b))
def recall(rows, k): at = [r for r in rows if r[0]]; return sum(r[k] for r in at) / len(at)
def far(rows, k): be = [r for r in rows if not r[0]]; return sum(r[k] for r in be) / len(be)
print(f"A recall={recall(data,1):.3f} false-alert rate={far(data,1):.3f}")
print(f"B recall={recall(data,2):.3f} false-alert rate={far(data,2):.3f}")
diffs = []
for _ in range(2000):
s = [rnd.choice(data) for _ in data]
if any(r[0] for r in s): diffs.append(recall(s, 2) - recall(s, 1))
diffs.sort()
print(f"recall difference B-A 95% bootstrap CI: [{diffs[50]:.3f}, {diffs[1949]:.3f}]")
Run with python3 lab_ab.py. Verified output:
A recall=0.650 false-alert rate=0.069
B recall=0.850 false-alert rate=0.037
recall difference B-A 95% bootstrap CI: [0.000, 0.388]
The test plan should be written before the replay is built, naming the attack behaviours, how many windows each gets and who labels them. Ideally the person who injects attacks did not design either rule set. Lab 6 stands in for this with seeded random outcomes, so its numbers show the method rather than the real performance of BWRP's rules.
6.2 Reading the result
B's recall is 0.85 against A's 0.65, and its false-alert rate is lower (3.7 % vs 6.9 %). The bootstrap interval for the recall difference runs from 0.000 to 0.388: the lower end touches zero. With only 40 attack windows, we cannot rule out that B is no better at recall than A, even though the point estimate favours it strongly. Each resampled window carries both rule sets' outcomes, so the bootstrap above already preserves the pairing; an exact test on the windows where A and B disagree would be the natural cross-check.
Report the counts beside the rates so a reader can judge the evidence: 34 of 40 against 26 of 40 attack windows, and 6 against 11 of 160 normal windows. With so few attack windows a handful of different outcomes would change the picture. That is the main reason Daniel asks for a larger and independently built replay before the next review.
6.3 Making the call
Leaders rarely need statistical certainty to act; they need to know the risk of each choice. B is cheaper in false alerts and probably better at recall. The downside of switching — if B were truly no better — is small. Daniel recommends switching, while expanding the attack-window set to 120 before the next review.
The decision is easier because B also produces fewer false alerts, which operators notice every shift. Daniel records the recommendation with its uncertainty, sets the next review date and names the colleague who will build the 120 attack windows. If the larger replay shows no recall advantage, the decision is revisited rather than defended.
6.4 Threats to validity
Injected attacks reflect the test author's imagination; real adversaries may differ. Replay omits operator reaction. Rule set B was designed after seeing some of these attack types, which favours it.
Each threat has a partial remedy. Ask a colleague who did not design B to write new attack windows. Note that replay measures alerts, not whether operators act on them. Record which attack types informed B's design and report B's results on those separately from the rest, so a reader can see whether its advantage holds on unseen behaviour.
Worked example (synthetic BWRP)
Daniel's evaluation note: "Replay R-6 (200 windows, 40 attack, seed 6). B: recall 34/40, false alerts 6/160. A: recall 26/40, false alerts 11/160. Recall difference 95 % bootstrap CI [0.000, 0.388]. Recommendation: adopt B; extend to 120 attack windows built by a colleague who did not design B."
Synthetic case — Bayside Water Reclamation Plant. The replay labels and alert outcomes are generated by the seeded script.
Errors and limits
Bootstrap intervals with few positives are unstable. The lab's alert outcomes are random draws, standing in for real rule behaviour.
Exercises
- EX026 — Counts behind the rates. Convert A's false-alert rate of 0.069 into a count of the 160 normal windows. Rubric: 3 = 11 of 160; 2 = ≈11; 1 = 0.069 × 200; 0 = none.
- EX027 — Interpreting a CI touching zero. What does a lower bound of 0.000 tell the decision-maker? Rubric: 3 = data consistent with no recall improvement; uncertainty from 40 positives; 2 = partial; 1 = "B is better"; 0 = none.
- EX028 — Why replay the same windows? Explain the benefit of a paired design here. Rubric: 3 = removes week-to-week variation; differences due to rules; 2 = partial; 1 = vague; 0 = none.
- EX029 — Independent attack authoring. Why should a colleague build the new attack windows? Rubric: 3 = avoids design bias in favour of B; 2 = partial; 1 = workload; 0 = none.
- EX030 — Asymmetric downside. Argue why switching to B is reasonable despite the interval. Rubric: 3 = lower false alerts and likely better recall; small cost if equal; reversible; 2 = partial; 1 = "statistically proven"; 0 = none.
Synthesis
A controlled evaluation pairs treatments on identical labelled data, reports counts and intervals, and lets leaders act proportionately on uncertain but useful evidence.
Continue studying — chapter 06 of 12
← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →