◇ OpenCyber Academy
Doctoral research · Chapter 5 · English

Chapter 5 — Main study and independent validation

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 5 — Main study and independent validation

Lesson map

Lesson 05 published. Map, example, EX027.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["5.1 Paired design"]
  N2["5.2 Three kinds of uncertainty"]
  N3["5.3 Independent validation on Testbed T2"]
  N4["5.4 The blinded operator review"]
  N5["5.5 Writing mixed results honestly"]
  N6["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX027"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["5.1 Paired design"]
  N2["5.2 Three kinds of uncertainty"]
  N3["5.3 Independent validation on Testbed T2"]
  N4["5.4 The blinded operator review"]
  N5["5.5 Writing mixed results honestly"]
  N6["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX027"]

[Learning outcomes and connection]
    |
    v
[5.1 Paired design]
    |
    v
[5.2 Three kinds of uncertainty]
    |
    v
[5.3 Independent validation on Testbed T2]
    |
    v
[5.4 The blinded operator review]
    |
    v
[5.5 Writing mixed results honestly]
    |
    v
[Chapter synthesis]
    |
    v
[Worked example] --> [Exercise EX027]

Learning outcomes and connection

After this chapter you can (1) design a paired experiment with common random numbers, (2) report uncertainty that separates simulation noise from model uncertainty, (3) validate on an independent second context, (4) incorporate a blinded operator review, and (5) write an honest results paragraph including mixed results. It executes the plan piloted in Chapter 4 and produces the claims Chapter 6 will attack.

5.1 Paired design

When comparing two detector settings, run both on the same random seeds (common random numbers, a standard simulation technique). The difference per seed removes shared noise and narrows confidence intervals. Lab 5 compares k = 0.8 with k = 0.6 on two plant variants: Plant A with an attack inflow gain of 0.05 and Plant B, a larger tank where the same pump manipulation produces only 0.02 units per step.

# Lab 5 (synthetic): paired comparison of detection delay on two plant models with common random numbers + bootstrap CI.
import random
def delay(seed, k, gain):
    rng = random.Random(seed); true, est, s = 50.0, 50.0, 0.0
    for t in range(700):
        inflow = 1.0 + rng.gauss(0, 0.05) + (gain if t >= 300 else 0.0)
        true += inflow - 0.02*true
        lvl = 50.0 + rng.gauss(0, 0.3) if t >= 300 else true + rng.gauss(0, 0.3)
        est += inflow + rng.gauss(0, 0.02) - 0.02*est
        s = max(0.0, s + abs(lvl - est) - k)
        if s > 6.0 and t >= 300: return t - 300
    return 400
for plant, gain in (("A", 0.05), ("B", 0.02)):
    diffs = [delay(i, 0.8, gain) - delay(i, 0.6, gain) for i in range(200)]
    rng = random.Random(7); boots = sorted(sum(rng.choice(diffs) for _ in diffs)/len(diffs) for _ in range(2000))
    print(f"plant {plant}: mean delay(k=0.8)-delay(k=0.6)={sum(diffs)/len(diffs):.1f} steps, 95% CI [{boots[50]:.1f}, {boots[1949]:.1f}]")

Expected output:

plant A: mean delay(k=0.8)-delay(k=0.6)=6.9 steps, 95% CI [6.5, 7.4]
plant B: mean delay(k=0.8)-delay(k=0.6)=45.4 steps, 95% CI [38.4, 53.0]

The robustness chosen in Chapter 4 costs about 7 steps on Plant A but about 45 steps on Plant B, where the attack signal is weaker. Undetected runs are capped at 400 steps (right-censoring); the Plant B mean is therefore a lower bound on the true cost. The bootstrap interval describes simulation noise only — it says nothing about whether the tank model represents any real plant.

5.2 Three kinds of uncertainty

Separate stochastic uncertainty (seed-to-seed variation, captured by the interval), parametric uncertainty (orifice coefficient, noise levels, calibration bias — explored by sensitivity analysis as in Lab 6) and structural uncertainty (is a single-tank mass balance the right model at all?). Only the second testbed addresses the third.

In the Setpoint Study the interval in Lab 5 covers only the first kind. Lab 6 covers part of the second, through calibration bias and allowance. Testbed T2 is the first evidence on the third, and its lower recall shows that the model's honest, lag-free flow meter was too simple. Report each kind in its own place so readers do not mistake one for another.

5.3 Independent validation on Testbed T2

T2 differs from the Plant A model in ways that matter: two coupled tanks, a different flow meter with its own lag, and a PLC scan cycle of 100 ms. Thresholds are frozen from Plant A tuning before any T2 data are examined — the validation would otherwise be another tuning exercise. Synthetic T2 results: class A replay detected in 18 of 20 episodes (two missed because the flow-meter lag hid small manipulations), median delay longer than in simulation, zero false alarms over 40 nominal hours, and class B evasion confirmed in all 10 trials. The study thus supports H1 with a qualification (lower recall on a laggy instrument) and confirms H4.

5.4 The blinded operator review

Ten synthetic operators (O01–O10) reviewed alarm timelines without knowing which detector produced them. Eight preferred the CUSUM alarm because it arrived with a residual trend they could interpret; two found a "residual" abstract and wanted the predicted level shown on the HMI. The finding is about interpretability, not detection accuracy, and it changes the design: the thesis recommends displaying predicted versus reported level.

Ten participants cannot support a statistical claim about operator populations, and the study does not make one. The review shows that interpretability is a design concern worth measuring with real consenting control-room staff later. Record the protocol, the blinding method and each operator's stated reason, so the recommendation can be traced to what was said rather than to a count.

5.5 Writing mixed results honestly

"In the synthetic Plant A model and on the synthetic Testbed T2, a mass-balance residual CUSUM detected class A sensor-replay attacks that a ±1.5 band alarm missed, with no false-alarm runs in nominal operation. Recall fell from 100% in simulation to 90% on T2 owing to flow-meter lag. Robustness to a 0.4-unit calibration bias required a larger allowance, which lengthened detection delay by about 7 steps on Plant A and at least 45 on the weaker-signal Plant B. When the attacker also controlled the flow meter (class B), detection failed in every trial, as predicted."

5.6 Common errors and limits

Independent seeds for paired comparisons; tuning on the validation testbed; hiding censored runs; presenting intervals as real-world uncertainty; omitting the failure condition from the abstract.

Each item has a concrete check in this study: Lab 5 uses common random numbers, T2 thresholds are frozen from Plant A tuning, capped runs are reported as censored at 400 steps, the bootstrap interval is labelled simulation noise, and the abstract states that class B evaded detection in every trial. A reviewer should be able to confirm each check from the archive.

Exercises — chapter 5

Chapter synthesis

Paired designs sharpen comparisons; uncertainty has three kinds and intervals capture only one; validation means frozen thresholds on an independent context; and an honest results paragraph carries its qualifications and its failure condition.

Continue studying — chapter 05 of 08

← Chapter 04 (draft) · Course chapter index · Chapter 06 (draft) →