Chapter 33 — Research methods II
Lesson map
Lesson 33 published. Map, example, EX097.
flowchart TD
N0["Learning outcomes"]
N1["33.1 Pre-registration"]
N2["33.2 Sample size for a proportion"]
N3["33.3 Effect sizes"]
N4["33.4 Replication package"]
N5["33.5 Negative results"]
N6["33.6 Version control as a lab notebook"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX097"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["33.1 Pre-registration"]
N2["33.2 Sample size for a proportion"]
N3["33.3 Effect sizes"]
N4["33.4 Replication package"]
N5["33.5 Negative results"]
N6["33.6 Version control as a lab notebook"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX097"]
[Learning outcomes]
|
v
[33.1 Pre-registration]
|
v
[33.2 Sample size for a proportion]
|
v
[33.3 Effect sizes]
|
v
[33.4 Replication package]
|
v
[33.5 Negative results]
|
v
[33.6 Version control as a lab notebook]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX097]
Learning outcomes
You will (1) design a replicable simulation study with pre-registered hypotheses; (2) choose sample sizes for proportion estimates; (3) report effect sizes and intervals rather than only significance; and (4) write a threats-to-validity section that a reviewer will accept.
33.1 Pre-registration
Writing hypotheses, measures, analysis and stopping rules before running the experiment prevents tuning the analysis to the result. For a detector study: "H1: CUSUM's false-alarm proportion over 300 quiet runs will be lower than the band's; primary measure: runs with any false alarm; analysis: difference in proportions with 95 % interval."
In Maya's capstone the stopping rule is the fixed seed count: 300 quiet and 300 attack runs, with no extra runs added if the first result looks weak. Fixing the measure, runs with any false alarm, also stops her switching later to a different measure that happens to favour CUSUM.
33.2 Sample size for a proportion
To estimate a proportion near p with 95 % margin of error E, n ≈ 1.96² p(1−p)/E². For p = 0.2 and E = 0.05, n ≈ 246. To claim a rate below 1 % when none are observed, the rule of three needs n ≈ 300.
Maya's 300 quiet seeds follow from the second formula: with 0 false alarms in 300 runs the upper bound is about 1 %, which is the claim she wants to be able to make. Forty runs, as in Chapter 25, could only support about 8.8 %. Her 300 attack seeds per slope come from the first formula, sized so that a detection proportion near 0.8 has a margin of about 5 points.
33.3 Effect sizes
"p < 0.05" says little about size. Report the difference: band 22.5 % vs CUSUM 0 % false-alarm runs, difference 22.5 points (interval computed). Readers can then decide whether it matters operationally.
In Chapter 25's numbers the band raised false alarms in 9 of 40 quiet runs and CUSUM in 0 of 40. The difference, 22.5 points, matters to operators because it means a nuisance alarm roughly one run in four instead of none observed. A p-value would say only that the difference is unlikely to be chance, which is the least interesting fact about it. An operator deciding whether to accept CUSUM on HMI-1 needs the size, and the bounds on each side.
33.4 Replication package
Publish code, seeds, parameters, Python version and the exact command. In this book, labs embed their verified output so a reader can check a rerun matches.
Maya's replication package for the detector study contains lab_detect.py, the seed ranges (0–39 quiet and 1000–1039 attack for the Chapter 25 run; 0–299 and 1000–1299 planned for the capstone), the threshold and slack values, the Python version and the command line. Anyone rerunning it should get 9/40 and 0/40 false-alarm runs exactly; a different count means something in the package is wrong, not that the result is uncertain.
Worked example (synthetic LDHP)
Maya's capstone pre-registration fixes: 300 quiet and 300 attack seeds, three attack slopes, threshold chosen on a separate tuning set of 50 seeds and then frozen. Primary outcome: runs with any false alarm, the measure fixed in H1; detection within 60 samples is a secondary outcome. She also states what result would falsify H1.
Synthetic case — Lakeside District Heating Plant. The plan is for a simulated study.
33.5 Negative results
If CUSUM does not beat the band at the frozen threshold, that is a publishable finding in a bachelor's thesis provided the method was sound. Report it with the same intervals you would have used for a positive result, and discuss which assumption — noise model, attack slope or threshold choice — most likely explains it. Hiding a negative result and re-running with new settings until something "works" is the practice pre-registration exists to prevent.
Maya's frozen threshold is what makes such a result credible.
33.6 Version control as a lab notebook
Commit the pre-registration file before the first experimental run, and commit results with the exact command that produced them. The commit history then shows the order of decisions, which is what an examiner checks when a result looks surprisingly clean.
For Maya, the first commit holds the pre-registration with the 300-seed design and the frozen-threshold rule; the tuning commit, on the separate 50 seeds, comes next; the evaluation commit after it records the command and outputs. An examiner can then confirm that the threshold existed before any evaluation seed was run.
Errors and limits
Simulation studies replicate easily but may not transfer to plants. Multiple comparisons inflate false discoveries; adjust or pre-declare a primary outcome.
Exercises
- EX097 — Sample size for 10 % ± 3 %. Compute n to estimate a proportion near 0.10 with ±0.03 at 95 %. Rubric: 3 ≈ 385; 2 = method; 1 = uses E = 3; 0 = none.
- EX098 — Why freeze thresholds? Explain why the threshold is tuned on separate seeds. Rubric: 3 = avoids overfitting to evaluation data; unbiased estimate; 2 = partial; 1 = vague; 0 = none.
- EX099 — A falsifying outcome. State the result that would falsify H1. Rubric: 3 = CUSUM false-alarm proportion ≥ band's (interval includes or exceeds zero difference); 2 = partial; 1 = "if CUSUM fails"; 0 = none.
Synthesis
Rigour in applied OT research comes from pre-registration, adequate samples, effect sizes with intervals and complete replication packages.
Continue studying — chapter 33 of 40
← Chapter 32 (draft) · Course chapter index · Chapter 34 (draft) →