Chapter 3 — Study design, ethics and reproducibility
Lesson map
Lesson 03 published. Map, example, EX015.
flowchart TD
N0["Learning outcomes"]
N1["3.1 Design"]
N2["3.2 Ethics and data"]
N3["3.3 Tuning governance"]
N4["3.4 Reproducibility"]
N5["Analysis plan"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX015"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["3.1 Design"]
N2["3.2 Ethics and data"]
N3["3.3 Tuning governance"]
N4["3.4 Reproducibility"]
N5["Analysis plan"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX015"]
[Learning outcomes]
|
v
[3.1 Design]
|
v
[3.2 Ethics and data]
|
v
[3.3 Tuning governance]
|
v
[3.4 Reproducibility]
|
v
[Analysis plan]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX015]
Learning outcomes
You will (1) design a before-after study with controls; (2) secure ethics and data agreements; (3) plan reproducible analysis.
3.1 Design
Amara uses an interrupted time-series design: four weeks before tuning and four weeks after, measuring alerts per hundred gigabytes, precision and recall each week. A true randomised design is impossible because tuning applies to the whole sensor estate, so she strengthens inference by normalising for traffic, recording all other changes during the period and testing recall on the same labelled event set before and after.
The design compares alert volume and detection of key events before and after documented tuning, using the same replay set. Measures and analysis are fixed in advance.
3.2 Ethics and data
Alerts contain internal addresses, hostnames and sometimes usernames. Corvane exports alerts with addresses replaced by consistent pseudonyms and removes payload excerpts, which could contain personal data. Amara's ethics approval covers interviews with analysts and the use of pseudonymised alert data; no customer data are involved. Data remain on a Corvane-managed analysis machine she accesses remotely, and only aggregate results leave it.
Alerts contain addresses and hostnames that can identify staff. Amara works under a data agreement, uses synthetic replays where possible, and stores extracts securely.
3.3 Tuning governance
Every tuning change is proposed by Amara, reviewed by an analyst and approved by the security lead, then recorded with the rule identifier, the change and the reason. Changes that disable a rule require evidence that the protected service does not exist at Corvane. This governance prevents the study from weakening detection to improve its numbers.
Every tuning change is proposed, reviewed by a second analyst, and recorded with its reason, so no detection is removed without justification.
Changes can be reversed quickly.
3.4 Reproducibility
All analysis is scripted, with scripts and synthetic sample data in a repository. Each figure is generated from code, and the dissertation lists the script version for each table, so examiners can rerun the analysis on the synthetic data.
Reproducibility requires that someone else could repeat the analysis and get the same result. Amara keeps the rule set versions, replay set identifiers, scripts and settings in version control, records the environment used, and writes a short guide to rerunning each step. Where company data cannot be shared, she provides synthetic data with the same structure so the method itself can be checked.
Analysis plan
Amara writes the analysis plan before any post-tuning data exist. She will compare mean weekly alerts per hundred gigabytes before and after, report the weekly series rather than only averages, and compute precision and recall on the labelled set with the rules as configured before and after. She decides in advance that any lost detection will be investigated individually and reported, whatever the overall recall. Fixing these choices early protects the study from selecting whichever analysis happens to look best once results are visible.
Worked example
The design document lists measures, weeks, governance steps and the change register template, approved by the security lead.
Synthetic case. The design is synthetic.
Errors and limits
Before-after designs are vulnerable to coincident changes; record everything else that changes.
Exercises
- EX015 — Time-series design. Why use four weeks either side? Rubric: 3 = capture weekly variation;; 2 = more data;; 1 = vague;; 0 = none.
- EX016 — Payload removal. Why remove payload excerpts from exports? Rubric: 3 = may contain personal data;; 2 = privacy;; 1 = vague;; 0 = none.
- EX017 — Disabling evidence. What evidence is required to disable a rule? Rubric: 3 = service absent at Corvane;; 2 = justification;; 1 = vague;; 0 = none.
- EX018 — Governance purpose. How does tuning governance protect validity? Rubric: 3 = prevents gaming metrics by weakening detection;; 2 = oversight;; 1 = vague;; 0 = none.
- EX019 — Data location. Why keep data on a Corvane machine? Rubric: 3 = control and minimisation;; 2 = security;; 1 = vague;; 0 = none.
- EX020 — Coincident changes. Give an example of a coincident change to record. Rubric: 3 = sensor upgrade or traffic shift;; 2 = example;; 1 = vague;; 0 = none.
Synthesis
A governed, normalised before-after design with reproducible scripts is credible within its limits.
Continue studying — chapter 03 of 08
← Chapter 02 (draft) · Course chapter index · Chapter 04 (draft) →