Chapter 5 — Proposal and pilot
Lesson map
Lesson 05 published. Map, example, EX027.
flowchart TD
N0["Learning outcomes and connection"]
N1["5.1 Proposal skeleton"]
N2["5.2 Pilot purpose"]
N3["5.4 Proposal checklist before supervisor review"]
N4["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4
N4 --> EX["Worked example"]
EX --> Q["Exercise EX027"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["5.1 Proposal skeleton"]
N2["5.2 Pilot purpose"]
N3["5.4 Proposal checklist before supervisor review"]
N4["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX027"]
[Learning outcomes and connection]
|
v
[5.1 Proposal skeleton]
|
v
[5.2 Pilot purpose]
|
v
[5.4 Proposal checklist before supervisor review]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX027]
Learning outcomes and connection
After this chapter you can (1) write a proposal with hypotheses and abandonment conditions, (2) define go/no-go criteria for a rig pilot, and (3) keep pilot data out of hypothesis tests.
5.1 Proposal skeleton
Question; gap; hypotheses H1–H4 (including failure); attacker model classes A/B; systems (simulator, Rig R2); metrics; analysis plan; ethics and safety; timeline; abandonment conditions (rig cannot be made safe; simulator fails verification).
Supervisors read proposals for decidability: can a reader tell what evidence would support H1 and what would refute it? "CUSUM is better" is not decidable; "event recall ≥95% with mean delay ≤60 steps and false-alarm runs ≤ those of the band alarm" is. Put failure hypothesis H4 in the same table as H1–H3 so it cannot be quietly dropped later.
Also state what you will not do: no production systems, no product-specific exploit development, no claims about sectors outside water storage loops. Scope exclusions are as important as inclusions.
5.2 Pilot purpose
Feasibility only: Can R2 run replay safely? Does the historian log at 1 Hz? Does the float valve work? Predeclared go criteria: three consecutive runs without float-valve trip, log loss <1%, e-stop tested. Pilot data do not test H1–H4.
A pilot that "looks promising" tempts peeking. Resist by writing the go/no-go sheet before the first pilot run and by storing pilot traces in a directory the analysis code refuses to read. If the pilot fails, you have learned that the rig needs work — which is the point — not that the detector is bad. Document who holds stop authority and test the e-stop in every session before attack scripts run.
5.3 Common errors and limits
Using pilot runs to tune thresholds that later "validate" the same runs; no abandonment condition.
Keep pilot traces in a separate directory that the analysis code refuses to read, and record in the proposal that thresholds are frozen from simulator tuning seeds only. Write abandonment conditions in measurable terms: for example, the rig is abandoned if the float valve cannot be shown to trip before the drip tray fills. Without such conditions a failing study simply drifts. Review those conditions with the supervisor before the first pilot run.
5.4 Proposal checklist before supervisor review
Before you send the proposal, confirm: (1) H1–H4 have numbers and H4 is a failure condition you are willing to confirm; (2) the attacker model distinguishes class A from class B; (3) metrics include event recall, delay, false-alarm runs and overfill at alarm — not point-adjusted F1 alone; (4) safety bounds name an independent layer and a stop authority; (5) abandonment conditions exist; (6) the pilot cannot leak into validation because paths are separated. A supervisor who returns the proposal with "scope clearer" is giving you a gift; a supervisor who accepts a vague proposal is setting up a painful year.
Exercises — chapter 5
- EX027 — H1–H4 with trip-margin thresholds. Write H1–H4 with numeric thresholds justified by a stated trip margin. Rubric: 3 = H1–H4 numeric with trip-margin justification; 2 = hypotheses without justification; 1 = H1 only; 0 = none.
- EX028 — Go/no-go with safety and data rules. Write go/no-go criteria including one safety and one data-quality rule. Rubric: 3 = safety + data-quality go/no-go (float valve, log loss); 2 = one rule type; 1 = vague go; 0 = none.
- EX029 — Three abandonment conditions. List three abandonment conditions and what you would switch to studying if abandoned. Rubric: 3 = three abandonment conditions + fallback study; 2 = conditions without fallback; 1 = one; 0 = none.
- EX030 — Attacker-model table for Harbor Loop. Draft the attacker-model table (capabilities, knowledge, goal, out of scope). Rubric: 3 = capabilities/knowledge/goal/out-of-scope table for Harbor Loop; 2 = missing a column; 1 = prose only; 0 = none.
- EX031 — Why pilots stay out of validation. Explain why pilot traces must not enter the validation set. Rubric: 3 = pilot leakage into validation invalidates freeze; 2 = 'contamination' without freeze link; 1 = vague; 0 = none.
- EX032 — Risk assessment for operator interviews. Write a one-paragraph risk assessment for interviewing operators Q01–Q08. Rubric: 3 = risks for Q01–Q08 with mitigations (consent, anonymization, stop); 2 = risks without mitigations; 1 = one risk; 0 = none.
Chapter synthesis
A proposal that states how it can fail, and a pilot that only asks whether the study can run safely, protect the dissertation from silent failure modes.
Continue studying — chapter 05 of 08
← Chapter 04 (draft) · Course chapter index · Chapter 06 (draft) →