Chapter 18 — Research methods I

Chapter 18 — Research methods I

Lesson map

Lesson 18 published. Map, example, EX052.

flowchart TD
  N0["Learning outcomes"]
  N1["18.1 From worry to question"]
  N2["18.2 Design choices"]
  N3["18.3 Validity"]
  N4["18.4 Reading the literature"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX052"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["18.1 From worry to question"]
  N2["18.2 Design choices"]
  N3["18.3 Validity"]
  N4["18.4 Reading the literature"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX052"]

[Learning outcomes]
    |
    v
[18.1 From worry to question]
    |
    v
[18.2 Design choices]
    |
    v
[18.3 Validity]
    |
    v
[18.4 Reading the literature]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX052]

Learning outcomes

You will (1) turn a vague worry into a researchable question; (2) choose between descriptive, comparative and experimental designs; (3) define variables, measures and threats to validity; and (4) search for and evaluate peer-reviewed OT security literature.

18.1 From worry to question

"Are our detectors good?" is not a question you can answer. "On LDHP's synthetic supply-temperature data, does a CUSUM detector reach a lower false-alarm rate than a fixed band at equal detection rate for slow ramps?" is. It names the population (synthetic data), the comparison, the outcome measures and the attack class.

Maya's version of this question became her research plan: CUSUM against a fixed band on simulated supply-temperature data, 40 seeds per condition, with false alarms per run, detection within 100 samples and mean delay as measures, and the slow 0.08 °C-per-sample ramp as the attack class.

18.2 Design choices

Descriptive studies characterise what exists (inventory of protocols on a network). Comparative studies contrast groups or methods without controlling assignment. Experiments manipulate one variable — detector type — while holding others fixed, typically through simulation in OT because experimenting on live plants is unsafe.

Maya's detector study is an experiment in this sense: detector type is the manipulated variable, and noise level, attack slope and seeds are held fixed across conditions. Running it on PLC-H itself would mean deliberately drifting the supply temperature of a live district heating network, which no permit would allow.

18.3 Validity

Internal validity asks whether the measured difference is caused by the manipulated variable; external validity asks whether results generalise beyond the synthetic plant. Construct validity asks whether "false-alarm rate per run" really measures operator burden. Name each threat and what you did about it.

Maya named two threats in her plan: the noise is Gaussian, which real supply-temperature sensors may not be (external validity), and she tuned the thresholds by hand (internal validity). Her capstone later addresses the second by tuning on a separate set of 50 seeds and freezing the threshold.

18.4 Reading the literature

Giraldo et al. (2018, ACM Computing Surveys, doi:10.1145/3203245) survey physics-based attack detection in CPS and highlight that many papers evaluate detectors without a consistent adversary model. Urbina et al. (CCS 2016, doi:10.1145/2976749.2978388) propose evaluating detectors by the impact a stealthy attacker can still cause while remaining undetected, rather than by detection rate alone. These two papers frame the detector lab in Chapter 25.

For Maya's study, Urbina et al.'s question becomes: how slow a ramp can stay below CUSUM's threshold, and how far could the supply temperature drift before it alarms? That is why she later halves the slope.

Worked example (synthetic LDHP)

Maya drafts: Question — the CUSUM vs band comparison above. Design — simulation experiment, 40 seeds per condition. Measures — false alarms per run, detection within 100 samples, mean delay. Threats — synthetic noise is Gaussian (external validity); thresholds tuned by hand (internal validity).

Synthetic case — Lakeside District Heating Plant. The research plan is a teaching exercise using simulated data only.

Errors and limits

Searching only vendor white papers yields marketing, not evidence. Do not cite a paper you have not opened; verify DOIs resolve.

Exercises

• EX052 — Sharpening a vague question. Rewrite "Is the IoT meter network secure?" as a researchable question. Rubric: 3 = population, measure and comparison stated; 2 = two; 1 = still vague; 0 = none.
• EX053 — Naming a validity threat. Identify one external-validity threat in Maya's design and one mitigation. Rubric: 3 = Gaussian noise/single plant; add real-noise profiles or second model; 2 = threat only; 1 = internal threat; 0 = none.
• EX054 — Urbina's metric in one sentence. State the evaluation idea of Urbina et al. (2016). Rubric: 3 = measure the maximum impact an undetected (stealthy) attacker can achieve, not just detection rate; 2 = partial; 1 = wrong; 0 = none.

Synthesis

Research begins with a sharp question, a design that fits it, honest validity threats and sources you have actually read.

Continue studying — chapter 18 of 40

← Chapter 17 (draft) · Course chapter index · Chapter 19 (draft) →