◇ OpenCyber Academy
Master's · Chapter 3 · English

Chapter 3 — Study design, ethics and reproducibility

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 3 — Study design, ethics and reproducibility

Lesson map

Lesson 03 published. Map, example, EX015.

flowchart TD
  N0["Learning outcomes"]
  N1["3.1 Design"]
  N2["3.2 Confounding"]
  N3["3.3 Ethics"]
  N4["3.4 Reproducibility"]
  N5["3.5 Pre-registration"]
  N6["3.6 Measures of association"]
  N7["Data management plan"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX015"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["3.1 Design"]
  N2["3.2 Confounding"]
  N3["3.3 Ethics"]
  N4["3.4 Reproducibility"]
  N5["3.5 Pre-registration"]
  N6["3.6 Measures of association"]
  N7["Data management plan"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX015"]

[Learning outcomes]
    |
    v
[3.1 Design]
    |
    v
[3.2 Confounding]
    |
    v
[3.3 Ethics]
    |
    v
[3.4 Reproducibility]
    |
    v
[3.5 Pre-registration]
    |
    v
[3.6 Measures of association]
    |
    v
[Data management plan]
    |
    v
[Worked example] --> [Exercise EX015]

Learning outcomes

You will (1) design an observational study of change records; (2) address confounding; (3) obtain ethics approval and data agreements; (4) plan reproducible analysis.

3.1 Design

The study uses one year of Larkspur's change records for storage resources. Each change has an origin (console or IaC), a time, a resource and whether it produced public exposure. An observational design is appropriate because Larkspur cannot randomly assign how teams make changes. One year covers seasonal peaks such as sales periods, when urgent changes are more likely. Each record comes from Larkspur's change log, with exposure taken from the provider's configuration history, so neither the team nor Lucas decides after the fact which change caused an exposure. The unit of analysis is the change, not the bucket.

3.2 Confounding

Console changes may happen more often under time pressure, in data-science teams or at night. Lucas records team, hour and resource type so he can check whether the difference holds within these groups. If the console rate is higher only at night or only in one team, explanation A would be weaker than it looks. Recording these variables before analysis lets him check within groups rather than explaining away differences after seeing them. Dr Tanaka later adds identity breadth to the list, because console users might also hold broad rights.

3.3 Ethics

Change records name engineers. The ethics application commits to pseudonymising engineer identities before analysis, reporting only by team and origin, and never using the study to assess individuals. Larkspur's data protection officer signs the data agreement. The ethics board added a condition: teams with fewer than five engineers are merged into an "other" group, so nobody can be identified by elimination. The pseudonymisation key stays at Larkspur, and Lucas never sees real names. The merging rule applies to every table and figure in the thesis, including the appendix breakdowns by account.

3.4 Reproducibility

Analysis scripts are written before the full data arrive, tested on synthetic records with the same structure and kept under version control. The synthetic generator is released with the thesis. Writing the scripts first means the analysis cannot be quietly shaped to the data. Testing them on synthetic records with Larkspur's structure also caught format problems before the real export arrived. Releasing the generator lets examiners and other students run the full analysis without access to Larkspur's records, which stay inside the data agreement.

3.5 Pre-registration

Lucas writes his hypotheses, primary comparison and planned subgroup checks in a dated document shared with Dr Tanaka before receiving the data. The document names the primary comparison, console against IaC exposure rates, the subgroup checks by team and hour, and the tests for clustering by template and identity breadth. When the pilot later forced a change to the definition of exposure, the change was logged against this document with its date and reason, before any main data were seen.

3.6 Measures of association

The primary result is the proportion of changes causing exposure by origin, the risk ratio, and a chi-square test; subgroup results follow. The risk ratio is the figure Larkspur's managers can use directly: how many times more likely a console change is to cause exposure. The chi-square test shows whether the difference could plausibly be chance at this sample size. Lucas checks before analysis that expected counts are large enough for the test to be valid, and plans to report the counts behind every percentage.

Data management plan

The data management plan states where pseudonymised records are stored (a university-approved encrypted store), who can access them (Lucas and Dr Tanaka), how long they are kept (until the degree is awarded plus one year) and how they are deleted. Larkspur keeps the key linking pseudonyms to engineers and never shares it.

Worked example

The ethics board asks how Lucas will avoid identifying an engineer in a small team. He agrees to merge teams with fewer than five engineers into an "other" group in all reports.

Synthetic case. Records and approvals are synthetic.

Errors and limits

Observational data show association; explanations need careful argument and subgroup checks.

Exercises

Synthesis

Design for confounders, protect people and fix the analysis in advance.

Continue studying — chapter 03 of 08

← Chapter 02 (draft) · Course chapter index · Chapter 04 (draft) →