◇ OpenCyber Academy
Master's · Chapter 3 · English

Chapter 3 — Study design, ethics and reproducibility

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 3 — Study design, ethics and reproducibility

Lesson map

Lesson 03 published. Map, example, EX015.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["3.1 Design matches question"]
  N2["3.2 Pre-data simulation"]
  N3["3.3 Ethics and safety"]
  N4["3.4 Reproducibility plan"]
  N5["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX015"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["3.1 Design matches question"]
  N2["3.2 Pre-data simulation"]
  N3["3.3 Ethics and safety"]
  N4["3.4 Reproducibility plan"]
  N5["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX015"]

[Learning outcomes and connection]
    |
    v
[3.1 Design matches question]
    |
    v
[3.2 Pre-data simulation]
    |
    v
[3.3 Ethics and safety]
    |
    v
[3.4 Reproducibility plan]
    |
    v
[Chapter synthesis]
    |
    v
[Worked example] --> [Exercise EX015]

Learning outcomes and connection

After this chapter you can (1) choose a design that matches the question, (2) simulate the design's bias and power before collecting data, (3) apply Menlo principles to industrial research, and (4) write a pre-analysis plan with data tiers.

3.1 Design matches question

Harbor Loop compares two detectors under controlled attacks in a model and a rig — a controlled experiment, not an observational study. If you asked "how often do plants see replay attacks?", you would need a survey with a sampling frame and a disclosure model, because plants that never report are invisible in public catalogues. If you asked how operators trust alarms, you would need interviews with ethics review. Matching design to question is the most common master's failure mode: students borrow a method they like and then hunt for a question that fits. Write the question first.

A second matching problem is unit of analysis. Detector comparison operates on attack episodes. Architecture comparison operates on reachability graphs. Operator studies operate on people and shifts. Mixing units — counting packets as if they were episodes — recreates point-adjustment confusion. State the unit before you collect anything.

3.2 Pre-data simulation

Before any experiment, simulate. Lab 3 (below, reused as the main detector lab) already shows class A detection and class B failure. Running it before writing the proposal tells you the design can distinguish the hypotheses under the model assumptions. Pre-data simulation also reveals whether your planned sample size is pointless: if eighty attack seeds always produce the same binary outcome, you need sensitivity analysis on parameters that might change the outcome (allowance k, orifice coefficient, sensor lag) more than you need a thousand identical seeds.

Record the simulation as a design artifact: seed ranges, parameter grid, decision rules. When a supervisor asks "why eighty runs?", the answer is in that artifact.

3.3 Ethics and safety

Menlo: respect for persons (operators you interview), beneficence (bound every physical experiment with an independent safety layer — never the PLC under test), justice (small utilities bear patch costs), respect for law and public interest. Never test on production. Campus Rig R2's worst outcome is water in a drip tray; a mechanical float valve and a physical e-stop bound it.

Beyond Menlo, OT research inherits safety culture: job briefings before lab sessions, two-person rules for energizing pumps, and a written stop authority that overrides the researcher's wish to finish a run. Record near misses (float valve almost reached) with the same seriousness as failed hypotheses.

3.4 Reproducibility plan

Register metrics (event recall, delay, false-alarm runs per 80 nominal seeds, overfill at alarm), seeds, thresholds and analysis code before the validation runs. Assign data tiers T0–T3 (Chapter 7 of the doctoral companion expands this; here, at least separate public code from rig configurations).

Store the registered plan with a timestamp the supervisor can verify, such as a dated commit in the project repository. Include the frozen k and h values, the seed ranges for tuning and for evaluation, and the rule for what counts as a detection. Any later change goes into a deviations table with a date and a reason, and is reported in the dissertation.

3.5 Worked example and Lab 3

# Lab 3: residual CUSUM vs band on synthetic tank; report event recall, delay, FA/shift, overfill@alarm.
import random
def run(seed,atk,det,k=0.8,h=6.0,steps=800,onset=400):
    rng=random.Random(seed); true=est=50.0; s=0.0; alarm=None; at=None
    for t in range(steps):
        inflow=1.0+rng.gauss(0,0.05)+(0.05 if atk and t>=onset else 0)
        true+=inflow-0.02*true
        lvl=true+rng.gauss(0,0.3); fm=inflow+rng.gauss(0,0.02)
        if atk and t>=onset: lvl=50.0+rng.gauss(0,0.3)
        if atk=="B" and t>=onset: fm=1.0+rng.gauss(0,0.02)
        est+=fm-0.02*est
        hit=abs(lvl-50)>1.5 if det=="band" else False
        if det=="cusum":
            s=max(0.0,s+abs(lvl-est)-k); hit=s>h
        if hit and alarm is None: alarm=t; at=true-50
    return alarm,at
for det in ("band","cusum"):
    fa=sum(run(i,None,det)[0] is not None for i in range(80))
    for atk in ("A","B"):
        r=[run(i,atk,det) for i in range(80)]
        hits=[(a-400,ov) for a,ov in r if a is not None and a>=400]
        d=sum(h[0] for h in hits)/len(hits) if hits else float("nan")
        ov=sum(h[1] for h in hits)/len(hits) if hits else float("nan")
        print(f"{det:5s} class{atk} FA_runs={fa}/80 detected={len(hits)}/80 mean_delay={d:.1f} mean_overfill@alarm={ov:.2f}")

Expected output:

band  classA FA_runs=11/80 detected=0/80 mean_delay=nan mean_overfill@alarm=nan
band  classB FA_runs=11/80 detected=0/80 mean_delay=nan mean_overfill@alarm=nan
cusum classA FA_runs=0/80 detected=80/80 mean_delay=37.7 mean_overfill@alarm=1.38
cusum classB FA_runs=0/80 detected=0/80 mean_delay=nan mean_overfill@alarm=nan

The lab runs eighty seeds for each detector and attack class and counts false-alarm runs on the same eighty seeds without an attack. The band alarm never fires on either attack, because replay holds the reported level near 50, yet it still raises false alarms on 11 of 80 nominal runs. CUSUM detects all class A episodes and none of class B, when the flow meter is also spoofed. That single table already contains the main result and its boundary. Read it before any rig work begins.

3.6 Common errors and limits

Choosing design after seeing data; active polling called "passive"; using the PLC as the only safety bound.

Each error has a concrete guard. Register the design and its decision rules before collecting data. Describe any measurement that sends requests to a device as active, even when it only reads. On Rig R2 the float valve and the physical e-stop are the safety bound; the PLC under test is never relied on to stop an overflow it may be the cause of.

Exercises — chapter 3

Chapter synthesis

Design before data, simulate before commit, bound every physical experiment, and freeze the analysis plan.

Continue studying — chapter 03 of 08

← Chapter 02 (draft) · Course chapter index · Chapter 04 (draft) →