◇ OpenCyber Academy
Bachelor's · Chapter 25 · English

Chapter 25 — Laboratory I

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 25 — Laboratory I

Lesson map

Lesson 25 published. Map, example, EX073.

flowchart TD
  N0["Learning outcomes"]
  N1["25.1 The experiment"]
  N2["25.2 Interpreting the output"]
  N3["25.3 Honest reporting"]
  N4["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4
  N4 --> EX["Worked example"]
  EX --> Q["Exercise EX073"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["25.1 The experiment"]
  N2["25.2 Interpreting the output"]
  N3["25.3 Honest reporting"]
  N4["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX073"]

[Learning outcomes]
    |
    v
[25.1 The experiment]
    |
    v
[25.2 Interpreting the output]
    |
    v
[25.3 Honest reporting]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX073]

Learning outcomes

You will (1) run a reproducible detector experiment with fixed seeds; (2) compare a fixed band against a CUSUM detector on false alarms, detection and delay; (3) report results with denominators; and (4) connect results to the evaluation critique in Urbina et al. (2016).

25.1 The experiment

The lab simulates LDHP's supply temperature at 90 °C with Gaussian noise (σ = 1 °C). In attack runs, starting at sample 100 a slow ramp adds 0.08 °C per sample — a stealthy setpoint drift in the spirit of ATT&CK for ICS T0836. The band detector alarms when the reading leaves 90 ± 3.5 °C. CUSUM accumulates positive deviations minus a slack of 0.5 °C and alarms above 12.

"""Lab 25: compare a fixed band and a CUSUM detector on synthetic supply temperature."""
import random
def run(seed, attack):
    rnd = random.Random(seed); s = 0.0; band = cus = None
    for t in range(200):
        x = 90 + rnd.gauss(0, 1.0) + (0.08 * (t - 100) if attack and t >= 100 else 0)
        if band is None and abs(x - 90) > 3.5: band = t
        s = max(0.0, s + (x - 90) - 0.5)
        if cus is None and s > 12: cus = t
    return band, cus
fa_b = sum(run(s, False)[0] is not None for s in range(40)); fa_c = sum(run(s, False)[1] is not None for s in range(40))
det = [run(1000 + s, True) for s in range(40)]
db = [b for b, _ in det if b is not None and b >= 100]; dc = [c for _, c in det if c is not None and c >= 100]
print(f"band : false alarms {fa_b}/40, detections {len(db)}/40, mean delay {sum(b-100 for b in db)/max(1,len(db)):.1f}")
print(f"cusum: false alarms {fa_c}/40, detections {len(dc)}/40, mean delay {sum(c-100 for c in dc)/max(1,len(dc)):.1f}")

Run with python3 lab_detect.py. Verified output:

band : false alarms 9/40, detections 37/40, mean delay 28.8
cusum: false alarms 0/40, detections 40/40, mean delay 23.1

With σ = 1 °C, the band's ±3.5 °C limits sit at 3.5 standard deviations, so false alarms are rare per sample but not per 200-sample run.

25.2 Interpreting the output

On 40 quiet runs the band raised false alarms in 9 runs — with σ = 1 °C, a single sample beyond 3.5 σ is rare but over 200 samples per run it happens — while CUSUM raised none. Against the ramp, CUSUM detected all 40 runs with a mean delay of 23.1 samples; the band detected 37 of 40 (counting only alarms at or after sample 100) with a mean delay of 28.8. Because the band also fires on noise, some of its "detections" are lucky noise excursions.

Wilson bounds make the comparison concrete: the band's 9 false-alarm runs in 40 give roughly 12 % to 38 %.

25.3 Honest reporting

Write: "CUSUM 0/40 false-alarm runs (Wilson 95 % upper ≈ 8.8 %), 40/40 detections, mean delay 23.1 samples, seeds 0–39 quiet and 1000–1039 attack." Do not write "CUSUM is perfect". Urbina et al. would also ask how large a ramp an attacker could sustain below CUSUM's threshold — a smaller slope might evade it for far longer.

The band's line should be reported the same way: "band 9/40 false-alarm runs (Wilson 95 % interval about 12 % to 38 %), 37/40 detections at or after sample 100, mean delay 28.8 samples". Maya's follow-up, halving the slope to 0.04 °C per sample with the same seeds, answers Urbina's question for one point on the curve.

Worked example (synthetic LDHP)

Maya halves the attack slope to 0.04 °C/sample in a copy of the script and records the change in delay, keeping the seeds. This turns one result into a curve and shows the attacker's trade-off between speed and stealth.

Synthetic case — Lakeside District Heating Plant. All data are simulated; seeds make the output reproducible.

Errors and limits

Gaussian, independent noise is optimistic. Thresholds were set by hand on the same data family they are evaluated on.

Exercises

Synthesis

A laboratory result is a count over seeds with bounds and an attacker model; changing the attacker is the next experiment, not an afterthought.

Continue studying — chapter 25 of 40

← Chapter 24 (draft) · Course chapter index · Chapter 26 (draft) →