Chapter 1 — Advanced research integrity and methods
Lesson map
Lesson 01 published. Map, example, EX001.
flowchart TD
N0["Learning outcomes and connection"]
N1["1.1 The object of study: a control loop with consequ"]
N2["1.2 A ladder of claims"]
N3["1.3 Safety before integrity of results"]
N4["1.4 Ethics and dual use"]
N5["1.5 Data that describes a plant is sensitive"]
N6["1.6 Metrics that resist gaming"]
N7["1.8 Practical activity — Lab 1: point adjustment inf"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX001"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["1.1 The object of study: a control loop with consequ"]
N2["1.2 A ladder of claims"]
N3["1.3 Safety before integrity of results"]
N4["1.4 Ethics and dual use"]
N5["1.5 Data that describes a plant is sensitive"]
N6["1.6 Metrics that resist gaming"]
N7["1.8 Practical activity — Lab 1: point adjustment inf"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX001"]
[Learning outcomes and connection]
|
v
[1.1 The object of study: a control loop with consequ]
|
v
[1.2 A ladder of claims]
|
v
[1.3 Safety before integrity of results]
|
v
[1.4 Ethics and dual use]
|
v
[1.5 Data that describes a plant is sensitive]
|
v
[1.6 Metrics that resist gaming]
|
v
[1.8 Practical activity — Lab 1: point adjustment inf]
|
v
[Worked example] --> [Exercise EX001]
Learning outcomes and connection
After this chapter you can: (1) explain why the physical process, not the packet, is the object of OT security research; (2) place a research claim on a ladder from network observation to physical consequence; (3) design an experiment whose worst credible outcome is bounded by an independent safety layer; (4) apply the Menlo principles to industrial research; and (5) choose detection metrics that cannot be gamed by scoring conventions. The chapter builds on your entry knowledge of research design and prepares the frontier mapping of Chapter 2.
1.1 The object of study: a control loop with consequences
An information system fails by disclosing, corrupting or withholding data. An industrial control system can fail in all of those ways, but its defining failure is different: a physical process leaves its safe envelope. A pump runs dry, a tank overflows, a chlorine dose exceeds its limit, a turbine overspeeds. NIST SP 800-82 Rev. 3 frames OT as systems that "interact with the physical environment" and stresses that safety and availability usually dominate the priority order that confidentiality leads in IT. That difference reshapes research. A detection method that flags every malicious Modbus write but cannot say whether the process was endangered has answered an IT question in an OT setting.
The doctoral researcher therefore models the loop explicitly. A sensor measures a process variable (level, pressure, flow). A controller, typically a programmable logic controller (PLC), compares it with a setpoint and computes a command. An actuator (pump, valve, breaker) changes the process. A supervisory layer (SCADA, HMI, historian) observes and adjusts setpoints. Beneath them all sits a safety instrumented system (SIS) or hard-wired interlock designed to bring the process to a safe state independently of the control system. Each element is a possible point of manipulation, and each element's honesty is something your detector either assumes or tests.
1.2 A ladder of claims
Write every claim on a five-rung ladder and refuse to climb a rung without evidence for it:
- Protocol observation — a frame with function code 6 (write single register) reached unit 1 (the Modbus specification defines the function codes).
- Control inference — the setpoint register changed from 50 to 80.
- Process inference — the tank level trajectory was consistent with the new setpoint.
- Safety inference — the trajectory would (or would not) cross the high-high trip within the modelled horizon.
- Consequence claim — equipment damage, release or harm would (or would not) have occurred, given the independent protection layers.
Most published OT detection papers stop at rung 1 or 2 and report accuracy; most practical value lies at rungs 3–5. A thesis need not reach rung 5, but it must say where it stops.
1.3 Safety before integrity of results
A badly run IT experiment loses data; a badly run OT experiment can injure someone. The integrity rules of ordinary research — honest reporting, pre-registration of analyses, retention of raw data — remain, but a new rule takes precedence: never test on production. Experiments run on simulations, on isolated laboratory rigs whose worst case is harmless, or, exceptionally, on an operator's system under the operator's written authorization, its own safety review and a named person with stop authority. Even in a laboratory the researcher should list the worst credible physical outcome of every experiment (for example "water spills onto the floor of the rig enclosure") and the independent layer that bounds it (a mechanical float switch, not the PLC under test). If the only protection is the system you are attacking, the experiment is not bounded.
1.4 Ethics and dual use
The Menlo Report adapts the Belmont principles to ICT research: respect for persons, beneficence, justice, and respect for law and public interest. In OT research beneficence dominates: an evasion technique that defeats every published detector is a real contribution and also a recipe. The ethical response is not silence but proportion. Publish the class of weakness, the defensive measurement and the mitigation; withhold vendor-specific exploitation detail until coordinated disclosure (Chapter 8) has run. Respect for persons applies to operators you interview or observe: they may reveal unsafe practices that could cost them their jobs, so consent forms must state who sees raw notes. Respect for law covers export control of some industrial technologies and contractual limits on testbed data.
1.5 Data that describes a plant is sensitive
Network captures, PLC programs, piping and instrumentation diagrams, and alarm logs together form a map of how to hurt a facility. Treat them as sensitive even when the facility is a university rig, because rigs reuse commercial controllers and their configurations reveal product behaviour. Define tiers early: T0 public, T1 shareable under licence, T2 shareable with vetted researchers, T3 never leaves the owner. Chapter 7 turns these tiers into a release manifest.
Assign each artefact a tier when it is created, not when the thesis is written, and record the owner who can change it. A capture taken on Testbed T2 starts at T2 even if it looks harmless. Lowering a tier needs a written reason, such as removing addresses and controller identifiers, and a second person who checks the result.
1.6 Metrics that resist gaming
Detection research in OT has a measurement problem. Attacks in time-series datasets are long contiguous segments; a widely used scoring convention, point adjustment, marks a whole attack segment as detected if any single point inside it is flagged. As Lab 1 shows, it can turn a detector that finds 10% of attack points into one reporting near-perfect F1. Arp et al. list this family of pitfalls — inappropriate performance measures, base-rate neglect, lab-only evaluation — for machine learning in security generally. For OT the defensible metric set is: event-level recall (fraction of attack episodes detected), time to detection measured from attack onset, false alarms per operator shift (because alarm floods are themselves a safety hazard), and physical impact at detection (how far the process had moved when the alarm fired). The last metric is OT-specific and is the one most often omitted.
1.7 Worked example: the first protocol review of the Setpoint Study
The candidate's first protocol proposed replaying recorded attacks on Kestrel Plant A's live historian network "read-only". The review board raised three issues. First, read-only is a claim, not a property: a capture tool on a span port is passive, but a tool that polls the PLC is active and adds load to a controller with a fixed scan cycle. Second, the worst credible outcome was not stated. Third, the metric was point-adjusted F1. The revised protocol (a) moves all attack experiments to a simulation of Plant A and later to the isolated Testbed T2; (b) states the worst outcome on T2 as "upper tank overflows into the drip tray", bounded by a mechanical float valve independent of the PLC; (c) reports event recall, delay, false alarms per 8-hour shift and maximum overfill; and (d) names the laboratory manager as stop authority.
1.8 Practical activity — Lab 1: point adjustment inflates scores
Run this offline. It creates a 100-sample synthetic label sequence with two attack segments and a detector that flags five samples.
# Lab 1 (synthetic): how "point-adjusted" scoring inflates an OT anomaly detector.
labels = [0]*20 + [1]*30 + [0]*30 + [1]*10 + [0]*10 # two attack segments (100 samples)
alarms = [0]*100
for i in (5, 40, 85, 86, 87): # detector flagged five samples
alarms[i] = 1
def scores(lab, pred):
tp = sum(l and p for l, p in zip(lab, pred)); fp = sum((not l) and p for l, p in zip(lab, pred))
fn = sum(l and not p for l, p in zip(lab, pred))
prec = tp/(tp+fp) if tp+fp else 0.0; rec = tp/(tp+fn) if tp+fn else 0.0
return round(prec, 3), round(rec, 3), round(2*prec*rec/(prec+rec), 3) if prec+rec else 0.0
def point_adjust(lab, pred):
out = pred[:]; i = 0
while i < len(lab):
if lab[i]:
j = i
while j < len(lab) and lab[j]: j += 1
if any(pred[i:j]): out[i:j] = [1]*(j-i)
i = j
else: i += 1
return out
def events(lab, pred):
segs, i = [], 0
while i < len(lab):
if lab[i]:
j = i
while j < len(lab) and lab[j]: j += 1
segs.append((i, j)); i = j
else: i += 1
return sum(any(pred[a:b]) for a, b in segs), len(segs)
print("pointwise P/R/F1 :", scores(labels, alarms))
print("point-adjust :", scores(labels, point_adjust(labels, alarms)))
print("events detected :", events(labels, alarms))
Expected output:
pointwise P/R/F1 : (0.8, 0.1, 0.178)
point-adjust : (0.976, 1.0, 0.988)
events detected : (2, 2)
Interpretation: both episodes were detected (event recall 1.0), which is genuinely useful, but the point-adjusted F1 of 0.988 hides that the first alarm in segment one came 20 samples after onset. Report event recall with delay, never point-adjusted F1 alone.
The same five alarms produce three very different stories. Pointwise, the detector found four of forty attack samples and raised one false alarm. Point-adjusted, it looks almost perfect. By events, it found both episodes, one of them late. Only the third view, paired with delay, tells an operator what would have happened at the tank, so it is the one the Setpoint Study reports.
1.9 Common errors and limits
- Treating a protocol anomaly as a process anomaly (rung 1 presented as rung 3).
- Calling a passive capture "safe" while running active polling tools.
- Using the PLC under test as the only safety bound in a laboratory.
- Reporting one aggregate score without false alarms per shift.
- Assuming university rigs are not sensitive because they are small.
Each error has a guard. Write the rung next to every claim in your notes. Call any tool that sends requests to a controller active, whatever it reads. Name the independent layer that bounds every laboratory run, such as the float valve on T2. Report false alarms per eight-hour shift beside recall. Tier university rig data as you would plant data.
Exercises — chapter 1
Rubric for all exercises in this book: 3 = correct, scoped, evidence-labelled and honest about limits; 2 = correct with a minor gap in scope or evidence; 1 = partially correct or unlabelled claims; 0 = missing, unsafe or overclaiming.
- EX001 — Ladder claims for overflow slogan. Objective: classify claims. Input: the statement "Our detector stops tank overflows." Steps: rewrite it as one claim on each of rungs 1–4 of the ladder for a Modbus setpoint attack. Evidence: four scoped claims. Constraint: no rung-5 claim without protection-layer data. Rubric: 3 = four rungs with Modbus FC6/40001, setpoint 50→80, trajectory, modelled trip; no rung-5 without protection data; 2 = four rungs but thin process link; 1 = drops a rung or invents rung-5; 0 = slogan restated.
- EX002 — Bound the T2 pump experiment. Objective: bound an experiment. For an attack on Testbed T2's pump, state the worst credible outcome, the independent protection layer and the stop authority. Evidence: a three-row table. Rubric: 3 = overflow≈2 L, float valve+10 L tray, named manager with e-stop; PLC-only layer ≤1; 2 = three rows but weak independence; 1 = missing stop authority; 0 = unbounded.
- EX003 — Passive versus active collection. Objective: passive versus active. Classify five data-collection methods (span-port capture, PLC polling, historian export, engineering-workstation project upload, firmware dump) as passive or active and give one risk each. Rubric: 3 = five methods with passive/active and one risk each (span, poll, historian, project upload, firmware); 2 = four correct; 1 = two; 0 = none.
- EX004 — Menlo review of publishing evasion. Objective: Menlo analysis. Apply the four principles to publishing a new evasion technique against physics-based detectors. Evidence: one paragraph per principle and a publication decision. Rubric: 3 = Menlo four principles + publish class/defence, withhold product steps; 2 = principles without decision; 1 = one principle; 0 = none.
- EX005 — Tier ladder program through simulator. Objective: data tiers. Assign T0–T3 to: a PLC ladder program, an anonymized alarm-count table, a network capture with IP addresses, a P&ID, a synthetic simulator. Justify each. Rubric: 3 = ladder T2/T3, alarms T1, IP capture T2, P&ID T2/T3, simulator T0 with reasons; 2 = four correct; 1 = two; 0 = none.
- EX006 — Event delay versus point adjustment. Objective: metrics. Using Lab 1, compute event-level detection delay for each segment and explain why point adjustment hides it. Rubric: 3 = delays 20 and 5 samples + why point adjustment hides them; 2 = delays only; 1 = one delay; 0 = none.
- EX007 — Setpoint Study protocol headings. Objective: protocol. Draft the ten headings of a Setpoint Study protocol including stop and abandonment conditions. Rubric: 3 = ten headings including stop (float valve, restart, log loss>1%) and abandonment; 2 = headings without stop/abandon; 1 = partial list; 0 = none.
Chapter synthesis
OT research studies control loops whose failure is physical. Climb the claim ladder only with evidence, bound every experiment with an independent safety layer, treat plant data as sensitive, and choose metrics — event recall, delay, false alarms per shift, physical impact at detection — that a scoring convention cannot inflate.