◇ OpenCyber Academy
Doctoral research · Chapter 4 · English

Chapter 4 — Baseline replication and pilot

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 4 — Baseline replication and pilot

Lesson map

Lesson 04 published. Map, example, EX021.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["4.1 Terms"]
  N2["4.2 Diagnosing a failed reproduction: TankBench-21"]
  N3["4.3 Verifying the simulator"]
  N4["4.4 Tuning without leakage: Lab 6"]
  N5["4.5 A bounded pilot on Testbed T2"]
  N6["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX021"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["4.1 Terms"]
  N2["4.2 Diagnosing a failed reproduction: TankBench-21"]
  N3["4.3 Verifying the simulator"]
  N4["4.4 Tuning without leakage: Lab 6"]
  N5["4.5 A bounded pilot on Testbed T2"]
  N6["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX021"]

[Learning outcomes and connection]
    |
    v
[4.1 Terms]
    |
    v
[4.2 Diagnosing a failed reproduction: TankBench-21]
    |
    v
[4.3 Verifying the simulator]
    |
    v
[4.4 Tuning without leakage: Lab 6]
    |
    v
[4.5 A bounded pilot on Testbed T2]
    |
    v
[Chapter synthesis]
    |
    v
[Worked example] --> [Exercise EX021]

Learning outcomes and connection

After this chapter you can (1) distinguish reproduction, replication and re-implementation, (2) diagnose why a published OT detection result cannot be reproduced, (3) verify a process simulator before trusting it, (4) design a bounded pilot on an isolated rig, and (5) make a predeclared go/no-go decision. It takes the hypotheses of Chapter 3 and prepares the main study of Chapter 5.

4.1 Terms

The ACM artifact-badging policy distinguishes repeatability (same team, same setup), reproducibility (different team, same artifacts) and replicability (different team, own artifacts). In OT these differ sharply because plants, controllers and datasets are rarely shareable. Fix your vocabulary before claiming any of them.

For the Setpoint Study, rerunning Lab 3 on the same code and seeds is repeatability. A colleague running the archived code is reproducibility. The TankBench-21 work is a re-implementation from the paper's description, closer to replication. Saying so precisely keeps a reviewer from reading a re-scoring as independent confirmation of the original result.

4.2 Diagnosing a failed reproduction: TankBench-21

The synthetic benchmark TankBench-21 reports an F1 of 0.96 for a band-plus-rate alarm. The candidate's re-implementation gets 0.31. A disciplined diagnosis lists hypotheses and tests each: (a) scoring convention — the paper used point adjustment (Lab 1 shows this alone can explain a gap of this size); (b) threshold leakage — thresholds were chosen on the test split; (c) label alignment — attack labels start when the attack script launched, not when the process first deviated; (d) sampling — the paper downsampled to 10 s, smoothing noise. Re-scoring with point adjustment yields 0.94, inside a predeclared tolerance of ±0.05, so the reproduction "succeeds" — but it succeeds in reproducing an inflated metric. The thesis records both numbers and adopts event-level recall plus delay.

4.3 Verifying the simulator

Verification asks whether the model is built right; validation asks whether it is the right model (standard modelling vocabulary). Verification tests for the Lab 3 tank include: equilibrium at 50.0 with nominal inflow (analytical check); mass conservation over 1,000 steps without noise; residual mean near zero under no attack; and seeds producing identical traces on two machines. A detector evaluated on an unverified simulator is evaluated on an unknown system.

Write each verification test as code that fails loudly, and run the suite before every batch of experiments. If the equilibrium check fails after a change to the outflow term, no detector result from that version is usable until it passes. Keep the test output with the run records so an examiner can see that verification preceded the results it supports.

4.4 Tuning without leakage: Lab 6

The CUSUM has two parameters, allowance k and threshold h. Tune them on separate seeds from the evaluation seeds and examine robustness to model error. Lab 6 sweeps k and adds a constant 0.4-unit level-sensor calibration bias — a realistic mismatch between model and instrument.

# Lab 6 (synthetic): CUSUM drift k vs. false alarms and detection delay - the tuning trade-off reviewers ask about.
import random
def first_alarm(seed, attack, k, h=6.0, steps=600, onset=300, bias=0.0):
    rng = random.Random(seed); true, est, s = 50.0, 50.0, 0.0
    for t in range(steps):
        inflow = 1.0 + rng.gauss(0, 0.05) + (0.05 if attack and t >= onset else 0.0)
        true += inflow - 0.02*true
        lvl = true + rng.gauss(0, 0.3) + bias
        if attack and t >= onset: lvl = 50.0 + rng.gauss(0, 0.3) + bias
        est += inflow + rng.gauss(0, 0.02) - 0.02*est
        s = max(0.0, s + abs(lvl - est) - k)
        if s > h: return t
    return None
for bias in (0.0, 0.4):
    for k in (0.4, 0.6, 0.8, 1.0):
        fa = sum(first_alarm(i, False, k, bias=bias) is not None for i in range(100))
        d = [a-300 for a in (first_alarm(i, True, k, bias=bias) for i in range(100)) if a is not None and a >= 300]
        print(f"bias={bias} k={k} false_alarm_runs={fa}/100 detected={len(d)}/100 mean_delay={sum(d)/len(d) if d else float('nan'):.1f}")

Expected output:

bias=0.0 k=0.4 false_alarm_runs=0/100 detected=100/100 mean_delay=25.6
bias=0.0 k=0.6 false_alarm_runs=0/100 detected=100/100 mean_delay=30.9
bias=0.0 k=0.8 false_alarm_runs=0/100 detected=100/100 mean_delay=38.0
bias=0.0 k=1.0 false_alarm_runs=0/100 detected=100/100 mean_delay=46.1
bias=0.4 k=0.4 false_alarm_runs=97/100 detected=19/100 mean_delay=35.4
bias=0.4 k=0.6 false_alarm_runs=4/100 detected=99/100 mean_delay=46.1
bias=0.4 k=0.8 false_alarm_runs=0/100 detected=100/100 mean_delay=55.6
bias=0.4 k=1.0 false_alarm_runs=0/100 detected=100/100 mean_delay=66.5

With a perfect model the smallest k looks best. A small calibration bias makes k = 0.4 useless: 97 runs alarm before the attack starts, and only 19 alarms land after onset. The defensible choice is k = 0.8, which trades about 7 steps of delay (versus k = 0.6 without bias) for robustness. This is the kind of result reviewers expect to see before the main study.

4.5 A bounded pilot on Testbed T2

The pilot answers feasibility questions only: can the rig run the replay scenario safely, can the historian log at 1 Hz, and does the float valve stop overflow? Pilot data are not used for hypothesis tests. Go/no-go criteria are written in advance: go if three consecutive runs complete with the float valve never reached, log loss below 1%, and no unplanned controller restart; otherwise revise. The pilot plan also records who may stop a run and how (a physical emergency stop that cuts pump power, independent of the PLC).

4.6 Common errors and limits

Calling a re-scoring a replication; trusting a simulator because it "looks right"; tuning on evaluation seeds; using pilot data to test hypotheses; lacking an independent emergency stop.

Each error lets a result look better than it is. A re-scoring with point adjustment only confirms that the original convention was used. A plausible-looking trace is not a passed verification test. Seeds shared between tuning and evaluation leak information. Pilot runs answer feasibility, not hypotheses. And without a physical stop that cuts pump power independently of the PLC, the pilot is not bounded.

Exercises — chapter 4

Chapter synthesis

Reproduce baselines with a predeclared tolerance and diagnose gaps systematically; verify the simulator; tune on separate seeds and stress-test against model error; and pilot only for feasibility with independent physical stop authority.

Continue studying — chapter 04 of 08

← Chapter 03 (draft) · Course chapter index · Chapter 05 (draft) →