◇ OpenCyber Academy
Master's · Chapter 3 · English

Chapter 3 — Study design, ethics and reproducibility

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 3 — Study design, ethics and reproducibility

Lesson map

Lesson 03 published. Map, example, EX015.

flowchart TD
  N0["Learning outcomes"]
  N1["3.1 Observational design"]
  N2["3.2 Data sources"]
  N3["3.3 Ethics"]
  N4["3.4 Reproducibility"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX015"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["3.1 Observational design"]
  N2["3.2 Data sources"]
  N3["3.3 Ethics"]
  N4["3.4 Reproducibility"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX015"]

[Learning outcomes]
    |
    v
[3.1 Observational design]
    |
    v
[3.2 Data sources]
    |
    v
[3.3 Ethics]
    |
    v
[3.4 Reproducibility]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX015]

Learning outcomes

You will (1) design an observational study; (2) plan ethical data use; (3) prepare reproducible analysis.

3.1 Observational design

Tariq cannot assign control types to protocols, so he designs an observational study of forty protocols with published administrative arrangements over a fixed three-year window. For each protocol he records control type at the start of the window, size measured by value managed, team size, audit history and whether a key-related incident occurred. Observational designs cannot prove causation, but careful measurement of likely confounders lets him test whether associations survive adjustment. He writes each variable's definition before collecting any data, and records the window dates.

3.2 Data sources

Data come from public sources only: protocol documentation, public governance records, published audit reports and post-incident reports. Lumen Analytics provides timestamps of queued privileged actions from its monitoring records for the secondary aim, with client permission and with protocol names pseudonymised. Tariq writes a codebook for each variable, including how to classify mixed arrangements, such as multi-signature for upgrades but a single key for pausing, which he codes by the weakest control on any fund-affecting function. Sources are logged with retrieval dates.

3.3 Ethics

Although data are public, Tariq considers harm. Publishing a list of protocols with weak controls could direct attackers to them. His ethics approval requires that results be reported in aggregate, with no protocol identified, and that any currently live weakness he notices be reported privately to the team through their published security contact before the dissertation is submitted. The university's ethics committee approves the plan, noting the responsible disclosure step. Tariq also stores data on an encrypted university drive. He records the security contacts he would use, with dates.

3.4 Reproducibility

Tariq's analysis uses short Python scripts with only the standard library, kept under version control with the codebook and a data dictionary. Each figure in the dissertation is produced by a script, and a single command reruns them all. He keeps a log of decisions, such as reclassifying a protocol after reading its governance forum, with dates and reasons. Reproducible analysis lets examiners check every number and protects him from mistakes made by hand-editing tables. The repository is shared with Dr Larsen.

3.5 Power and limits

With forty protocols and an expected incident rate around one in five, Tariq's study has limited power to detect modest differences. He states this in advance and plans to report risk ratios with intervals rather than relying on significance tests. He also notes measurement limits: incidents may go unreported, and control types may change during the window. Stating limits in the design chapter shows examiners he understood them before seeing results. He also records the minimum difference he could plausibly detect, so readers can judge results fairly.

Worked example

The design covers 40 protocols over three years with five recorded variables, aggregate-only reporting, private disclosure of live weaknesses and scripted analysis.

Synthetic case. Protocols, data and approvals are synthetic.

Errors and limits

Publishing named lists of weakly controlled protocols could direct attackers to them.

Exercises

Synthesis

A careful observational design with confounders, ethical reporting and scripted analysis supports honest conclusions.

Continue studying — chapter 03 of 08

← Chapter 02 (draft) · Course chapter index · Chapter 04 (draft) →