Chapter 3 — Study design, ethics and reproducibility
Lesson map
Lesson 03 published. Map, example, EX015.
flowchart TD
N0["Learning outcomes"]
N1["3.1 Choosing a design"]
N2["3.2 Checking assumptions"]
N3["3.3 Ethics"]
N4["3.4 Reproducible analysis"]
N0 --> N1 --> N2 --> N3 --> N4
N4 --> EX["Worked example"]
EX --> Q["Exercise EX015"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["3.1 Choosing a design"]
N2["3.2 Checking assumptions"]
N3["3.3 Ethics"]
N4["3.4 Reproducible analysis"]
N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX015"]
[Learning outcomes]
|
v
[3.1 Choosing a design]
|
v
[3.2 Checking assumptions]
|
v
[3.3 Ethics]
|
v
[3.4 Reproducible analysis]
|
v
[Worked example] --> [Exercise EX015]
Learning outcomes
You will (1) choose a quasi-experimental design when randomisation is impossible; (2) obtain ethical approval for workplace security research; (3) fix an analysis plan before data collection.
3.1 Choosing a design
Harrowgate will not randomise staff, because the security team has already chosen three departments to pilot feedback. Priya therefore uses a difference-in-differences design, following Shadish, Cook and Campbell's guidance on quasi-experiments: compare the change in click rate in feedback departments with the change in comparison departments over the same period. The comparison removes influences shared by both groups, such as a seasonal rise in real phishing or a news story about fraud, which would otherwise be credited to feedback. Dr. Bello approves the design choice.
3.2 Checking assumptions
The design assumes both groups would have changed similarly without feedback. Priya examines four earlier quarterly simulations and finds that click rates in the two groups moved together, with neither trending faster. She also checks that simulation emails were identical across groups, because harder lures in one group would distort comparisons. Dr. Bello asks her to report these checks in full so examiners can judge the assumption, rather than asserting it is satisfied in a single sentence. She plots the four quarters for both groups.
3.3 Ethics
The study involves staff whose clicks might be seen as failures. The university ethics committee approves it on conditions: results are reported only at department level, individuals are pseudonymised by the security team before Priya receives data, no one is disciplined on the basis of study data and staff are told that simulations are part of training. Priya agrees and adds that the dissertation will avoid language blaming staff, consistent with the usability literature. The security team confirms in writing that it will apply the pseudonyms before export.
3.4 Reproducible analysis
Priya writes her analysis plan before collecting outcome data: primary outcome, click rate per department group; secondary outcome, report rate; estimator, difference-in-differences; uncertainty, a bootstrap interval from a thousand resamples with a fixed seed. She stores scripts in version control and labels any later analysis as exploratory. Dr. Bello signs and dates the plan. She notes that fixing the plan protects her from choosing analyses after seeing which one looks best. A copy of the signed plan is lodged with the programme office, and the security team receives one too, so every party knows which analyses are confirmatory.
Worked example
Priya uses difference-in-differences, checks parallel trends and identical lures, obtains approval with department-level reporting and no discipline, and freezes the analysis plan.
Synthetic case. Approval conditions are synthetic.
Errors and limits
Using simulation results to discipline staff undermines trust and reporting.
Exercises
- EX015 — Design chosen. Which design does Priya use? Rubric: 3 = difference-in-differences;; 2 = partly correct;; 1 = randomised;; 0 = none.
- EX016 — Shared influences. What does the comparison group remove? Rubric: 3 = influences shared by both groups;; 2 = partly correct;; 1 = vague;; 0 = none.
- EX017 — Assumption check. What did the four earlier simulations show? Rubric: 3 = rates moved together;; 2 = partly correct;; 1 = diverged;; 0 = none.
- EX018 — Lure check. Why check that emails were identical? Rubric: 3 = harder lures would distort comparison;; 2 = partly correct;; 1 = vague;; 0 = none.
- EX019 — Reporting level. At what level are results reported? Rubric: 3 = department level;; 2 = partly correct;; 1 = individual;; 0 = none.
- EX020 — Uncertainty method. How is uncertainty estimated? Rubric: 3 = bootstrap with a thousand resamples and fixed seed;; 2 = partly correct;; 1 = vague;; 0 = none.
Continue studying — lesson 03 (chapter 03 of 08)
← Chapter 02 · Course chapter index · Chapter 04 →
Continue to the next authored lesson.