◇ OpenCyber Academy
Master's · Chapter 3 · English

Chapter 3 — Study design, ethics and reproducibility

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 3 — Study design, ethics and reproducibility

Lesson map

Lesson 03 published. Map, example, EX015.

flowchart TD
  N0["Learning outcomes"]
  N1["3.1 Design choice"]
  N2["3.2 Ground truth for attacks"]
  N3["3.3 Ethics"]
  N4["3.4 Reproducibility"]
  N5["3.5 Threats to validity"]
  N6["3.6 Registration"]
  N7["Stopping rule"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX015"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["3.1 Design choice"]
  N2["3.2 Ground truth for attacks"]
  N3["3.3 Ethics"]
  N4["3.4 Reproducibility"]
  N5["3.5 Threats to validity"]
  N6["3.6 Registration"]
  N7["Stopping rule"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX015"]

[Learning outcomes]
    |
    v
[3.1 Design choice]
    |
    v
[3.2 Ground truth for attacks]
    |
    v
[3.3 Ethics]
    |
    v
[3.4 Reproducibility]
    |
    v
[3.5 Threats to validity]
    |
    v
[3.6 Registration]
    |
    v
[Stopping rule]
    |
    v
[Worked example] --> [Exercise EX015]

Learning outcomes

You will (1) choose a design suited to an operational intervention; (2) plan ethics and data protection; (3) make the study reproducible; (4) define ground truth for "suppressed attack".

3.1 Design choice

Random assignment of rules to tuning is possible but the SOC wants its noisiest rules fixed first. Tariq uses a controlled before-after design: ten tuned rules and ten comparable untuned rules, four weeks before and four after. Comparing changes in both groups removes effects common to all rules.

Matching the control rules on volume and ATT&CK tactic makes the comparison fairer: an untuned rule watching a quiet tactic would behave differently from a noisy one. Four weeks before and four after give each group its own baseline, so the analysis compares how much each group changed rather than their raw levels.

3.2 Ground truth for attacks

Real attacks are rare and unknown in advance. With the CISO's approval, the utility's authorised purple-team exercises inject benign simulated behaviour matching rule logic during the study, logged with exact times. A "suppressed attack" is an injected event that matched a rule but was removed by an allow-list.

Eight injected events in total is a small number, which Tariq states as a limit. Exact timing is what makes them usable: each event can be matched to the rule it should trigger and checked for whether an allow-list removed it. Because the exercise lead logs times independently of the SIEM, a suppressed event cannot be missed simply because no alert exists.

3.3 Ethics

Alerts contain user names. Tariq's data is pseudonymised before leaving the SOC; only the SOC sees real names. Ethics approval covers analyst interviews. The purple-team scope is signed by the CISO and excludes operational technology networks entirely.

Excluding operational technology networks matters at a water utility: simulated behaviour on control systems could affect treatment or pumping, and no study benefit would justify that. Pseudonymisation at source means that even if the export were lost, it would not show which analyst or employee triggered which alert.

3.4 Reproducibility

Allow-list entries are version-controlled with reasons; analysis scripts and counts are archived; the rule set is stored in Sigma format with tuning diffs.

Version control is what later makes the narrowing of entry AL-07 traceable: the change log shows the original host-account pair, the narrowed condition and the date. The tuning diffs in Sigma format let another SOC see exactly what was changed in each rule without access to Pelham's SIEM. Archiving the weekly counts per rule lets anyone rerun the difference-in-differences lab on the real study figures, not only on the synthetic ones.

3.5 Threats to validity

Seasonal change, log-source outages, other tuning by colleagues, and analysts' awareness of the study. Each has a planned check.

Each check is concrete. Seasonal change is absorbed by the control group. Log-source outages are checked against ingestion counts per source. Other tuning is checked in the rule change log. Analysts' awareness is harder: they knew the study was running, which might change how they handled alerts, so Tariq lists it as a limitation rather than claiming to have removed it.

3.6 Registration

The design, outcomes and checks are registered with the supervisor before week 1.

Registering before week 1 means that the outcomes, the comparison group and the stopping rule for a suppressed attack were fixed before any data arrived. When an injected event was suppressed in week 6, the registered rule dictated what happened next, a review within a day, rather than leaving Tariq to decide under pressure whether to pause the study. The registration also lists the eight planned purple-team behaviours by rule, so nobody could add easier ones later to improve the safety result.

Stopping rule

If any injected event is suppressed, the responsible allow-list entry is reviewed within a day; the study continues, but the finding is recorded and the entry may be narrowed.

Worked example

Tariq's design table: "Groups: T (10 tuned) and C (10 untuned, matched on volume and ATT&CK tactic). Weeks 1–4 baseline, 5–8 tuned. Primary: weekly alerts per group. Safety: injected purple-team events per rule (8 total) and how many were suppressed. Data: pseudonymised at source."

Synthetic case. The design is invented.

Errors and limits

Injected events test only behaviours that were injected; real attackers may differ.

Exercises

Synthesis

Operational studies need comparison groups, defined ground truth, protected data and registered threats to validity.

Continue studying — chapter 03 of 08

← Chapter 02 (draft) · Course chapter index · Chapter 04 (draft) →