◇ OpenCyber Academy
Master's · Chapter 3 · English

Chapter 3 — Study design, ethics and reproducibility

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 3 — Study design, ethics and reproducibility

Lesson map

Lesson 03 published. Map, example, EX015.

flowchart TD
  N0["Learning outcomes"]
  N1["3.2 Handling confounders"]
  N2["3.3 Consent and TLP"]
  N3["3.4 Personal data"]
  N4["3.5 Reproducibility plan"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX015"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["3.2 Handling confounders"]
  N2["3.3 Consent and TLP"]
  N3["3.4 Personal data"]
  N4["3.5 Reproducibility plan"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX015"]

[Learning outcomes]
    |
    v
[3.2 Handling confounders]
    |
    v
[3.3 Consent and TLP]
    |
    v
[3.4 Personal data]
    |
    v
[3.5 Reproducibility plan]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX015]

Learning outcomes

You will (1) choose between observational and quasi-experimental designs; (2) identify confounders in sharing studies; (3) obtain consent from sharing-community members for research use; (4) plan a reproducible analysis.

3.1 Observational limits

Lucía cannot randomly assign ports to CPSG membership. She can compare members with non-members, but members may differ — larger budgets, more staff — and those differences, not sharing, might explain faster blocking. A design that ignores this is a correlation study dressed as an evaluation.

Lucía's design responds with two comparisons: four ports that joined CPSG in 2025 are compared with themselves before and after joining, and members are compared with six volunteer non-members matched on size. Neither removes all differences, but they fail in different ways, so agreement between them is more convincing than either alone.

3.2 Handling confounders

Options include matching members to non-members of similar size, comparing the same port before and after joining, or adjusting for SOC staffing. Each choice is written into the proposal, not chosen after seeing which gives the nicer result.

Lucía chose both matching on size and a before-and-after stratum for the four ports that joined in 2025, and wrote that choice into the proposal before collecting any timings. If the before-and-after stratum shows a consistent drop on joining, she can speak more strongly than the cross-sectional comparison allows.

3.3 Consent and TLP

CPSG data carries TLP:AMBER. Research use needs explicit agreement from the community's governance board, and each member must be able to opt out. Results will be aggregated so that no single member's timings can be recovered. Non-member data must come from voluntary participants, not from probing their infrastructure.

The governance board approved Lucía's design with three conditions: results aggregated to groups of at least three, no port names, and deletion of raw timings after the dissertation is marked. The six non-member ports supplied their own timings voluntarily; nothing was collected from their infrastructure without their participation.

3.4 Personal data

Time-to-block logs may include analyst user names. Lucía removes them at source; she needs timings, not people. Data-protection law such as the GDPR applies even in security research, and minimisation is the simplest compliance step.

Removing names at source means the member's export to Lucía contains only timestamps, indicator type and decision. If an analyst's name were kept even in a working file, the data would identify the person who was slow to block a particular indicator, a harm to that analyst that the study does not need to risk.

3.5 Reproducibility plan

Analysis scripts in the standard library, a data dictionary, synthetic sample files and a fixed random seed for any resampling. The plan is part of the proposal.

For Lucía, the synthetic sample files use exactly the fields in her data dictionary, t_pub, t_block, ttb_h, ioc_type and decision, so the same scripts run on both, and an examiner without access to CPSG's AMBER data can still run every script. The fixed seed matters for the bootstrap interval around the medians; without it, a rerun would give a slightly different interval and look like an error.

Worked example

Lucía proposes a before-and-after design for four ports that joined CPSG in 2025, plus matched comparison with six non-members who volunteer. The governance board approves with conditions: aggregation to groups of at least three, no port names, and deletion of raw timings after the dissertation is marked.

Synthetic case. The design, ports and board conditions are invented.

Errors and limits

Small samples cannot support causal claims even with good design; frame findings accordingly. Consent by a board does not override an individual member's opt-out.

Exercises

Synthesis

Name confounders early, pick a design that weakens them, obtain board and member consent, minimise personal data, and plan reproducibility before data arrive.

Continue studying — chapter 03 of 08

← Chapter 02 (draft) · Course chapter index · Chapter 04 (draft) →