◇ OpenCyber Academy
Master's · Chapter 3 · English

Chapter 3 — Study design, ethics and reproducibility

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 3 — Study design, ethics and reproducibility

Lesson map

Lesson 03 published. Map, example, EX015.

flowchart TD
  N0["Learning outcomes"]
  N1["3.1 Design"]
  N2["3.2 Ethics"]
  N3["3.3 Reproducibility"]
  N4["3.4 Variables"]
  N5["3.5 Bias"]
  N6["Analysis plan contents"]
  N7["Data storage"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX015"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["3.1 Design"]
  N2["3.2 Ethics"]
  N3["3.3 Reproducibility"]
  N4["3.4 Variables"]
  N5["3.5 Bias"]
  N6["Analysis plan contents"]
  N7["Data storage"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX015"]

[Learning outcomes]
    |
    v
[3.1 Design]
    |
    v
[3.2 Ethics]
    |
    v
[3.3 Reproducibility]
    |
    v
[3.4 Variables]
    |
    v
[3.5 Bias]
    |
    v
[Analysis plan contents]
    |
    v
[Data storage]
    |
    v
[Worked example] --> [Exercise EX015]

Learning outcomes

You will (1) design an observational study; (2) address privacy and ethics; (3) plan reproducible analysis.

3.1 Design

Chloé uses six months of monthly compliance records for all 600 laptops, linked to device attributes: local administrator flag, location (remote or office), department and last policy refresh.

Six months of monthly checks gives six observations per laptop, enough to see drift appear after compliance rather than at deployment. Linking to existing attributes, instead of asking users questions, avoids recall bias and keeps the study observational. The last policy refresh date is the variable Explanation B depends on: a remote laptop that has not checked in recently cannot have received the baseline again.

3.2 Ethics

Records link devices to users. Chloé works with pseudonymised device IDs; department is the finest grouping reported; results are never used to assess individuals. Redbourne's ethics committee and Halvorsen's data protection lead approve the plan.

Halvorsen's works council asked whether results could be used to discipline staff. Chloé could answer by showing the approval terms: device-level data, pseudonymised identifiers and department-level reporting only. Because no result names a person, the study cannot be turned into a tool for monitoring individual employees, which also makes staff less likely to object to the extract being shared.

3.3 Reproducibility

The analysis plan is written before data access; code runs on the extract and produces every table.

Writing the plan first prevents choosing the analysis after seeing which comparison gives the most striking result. Chloé's plan names the three comparisons, the risk ratio and chi-square tests, and the stratification by admin status, all before the extract arrives. Because one script produces every table from the extract, an examiner can rerun it and get the same figures, which the second examiner later does.

3.4 Variables

The outcome is drift during the six months; candidate explanations are local admin, remote working and department.

The outcome is binary for each laptop: drifted at least once in the six months, or not. Each candidate explanation becomes a group comparison: admin versus none, remote versus office, engineering versus other departments. Coding them this way makes the three comparisons directly comparable in the same units, a risk ratio, which is what lets Chloé say which association is strongest. All three comparisons divide the same 600 laptops differently: 120 admin against 480 others, 200 remote against 400 office, and 150 engineering against 450 in other departments.

3.5 Bias

Devices replaced during the study are excluded and counted, so exclusion is visible.

Replaced laptops are excluded because they do not have six months of records. Counting them matters because replacement may not be random: if laptops with many problems are replaced first, excluding them could make the remaining fleet look better behaved. Reporting the count lets a reader judge whether the exclusion could change the result. The same rule applies to laptops whose user changed role mid-study: they stay in, but the role change is recorded.

Analysis plan contents

The plan lists each comparison, the statistic used, the threshold for significance and how missing values are handled. It also states what would count as evidence against each explanation. For example, if admin and non-admin devices drifted at similar rates, Explanation A would be weakened.

The plan is shared with Dr Mehta and timestamped in the university's repository before Chloé receives the full extract.

Data storage

The extract is stored on an encrypted university drive with access limited to Chloé and Dr Mehta, and deleted after the dissertation is examined, as the approvals require.

Consent and notice

Halvorsen informs staff through its usual privacy notice that device compliance data may be analysed in pseudonymised form for security improvement, and the works council is briefed before data are shared.

Worked example

Halvorsen's works council asks whether results could be used to discipline staff. Chloé shows the approval terms: device-level, pseudonymised, department-level reporting only.

Synthetic case. Synthetic.

Errors and limits

Pseudonymisation reduces but does not remove re-identification risk in small departments.

Exercises

Synthesis

Design for privacy and reproducibility from the start.

Continue studying — chapter 03 of 08

← Chapter 02 (draft) · Course chapter index · Chapter 04 (draft) →