Chapter 3 — Study design, ethics and reproducibility
Lesson map
Lesson 03 published. Map, example, EX015.
flowchart TD
N0["Learning outcomes"]
N1["3.1 Choosing the design"]
N2["3.2 Ethics and approval"]
N3["3.3 Reproducible plan"]
N4["3.4 Pilot of the reviewer"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX015"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["3.1 Choosing the design"]
N2["3.2 Ethics and approval"]
N3["3.3 Reproducible plan"]
N4["3.4 Pilot of the reviewer"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX015"]
[Learning outcomes]
|
v
[3.1 Choosing the design]
|
v
[3.2 Ethics and approval]
|
v
[3.3 Reproducible plan]
|
v
[3.4 Pilot of the reviewer]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX015]
Learning outcomes
You will (1) design an observational comparison with confounders in mind; (2) obtain ethical and organisational approval; (3) plan reproducible analysis.
3.1 Choosing the design
Tessaly has ten business units: five review access quarterly and five annually. Lucas cannot randomise frequency, so he designs a cross-sectional comparison. An independent reviewer samples 300 accounts from each group and judges each against the role catalogue. To address confounding, he records unit size, staff turnover and system count, because high turnover could raise excess access regardless of review frequency. He plans to compare rates and then check whether the difference holds within units of similar turnover. He also records the number of accounts in each unit.
3.2 Ethics and approval
Although no one is experimented on, the study handles data about employees' access. Lucas obtains approval from Marlow's ethics committee and Tessaly's data protection officer. Reviewers see account identifiers only, not names, and results are reported by group, never by person or by named unit. Findings will not be used to discipline anyone. Lucas also agrees with Tessaly that any serious excess access found, such as administrator rights for a departed contractor, will be reported to the security team immediately as an operational issue.
3.3 Reproducible plan
Lucas writes an analysis plan before seeing the data: the outcome, the comparison, a bootstrap interval for the difference in rates, the turnover check and how missing judgements will be handled. He fixes random seeds in scripts and stores the plan with a date. He notes that pre-specifying analyses protects against choosing the most flattering result after the fact, a problem Wohlin and colleagues (2012) discuss for software engineering experiments that applies equally to governance studies. All scripts and the plan live in one versioned folder for review.
3.4 Pilot of the reviewer
Before the main study, Lucas tests whether excess access can be judged consistently. Two reviewers independently judge 40 accounts. They agree on 36, and discussion reveals that most disagreements concerned shared mailboxes, which the role catalogue did not cover. Lucas adds a rule for shared resources. This pilot shows that the outcome is measurable but depends on catalogue quality, a limit he will report. It also suggests a practical benefit for Tessaly: a better catalogue makes every future review easier. He records the rule in the codebook.
Worked example
Lucas designs a 300-plus-300 account comparison, records turnover as a confounder, secures approval and pre-registers his analysis.
Synthetic case. All units, accounts and agreement figures are synthetic.
Errors and limits
Measured confounders do not remove unmeasured ones. Catalogue quality limits validity.
Exercises
- EX015 — Group sizes. How many accounts are sampled per group? Rubric: 3 = 300;; 2 = three hundred;; 1 = vague;; 0 = none.
- EX016 — Key confounder. Which confounder could raise excess access regardless of frequency? Rubric: 3 = staff turnover;; 2 = turnover;; 1 = vague;; 0 = none.
- EX017 — Approvals needed. Whose approval did Lucas obtain? Rubric: 3 = Marlow's ethics committee and Tessaly's data protection officer;; 2 = both;; 1 = vague;; 0 = none.
- EX018 — Serious finding rule. What happens if serious excess access is found? Rubric: 3 = reported immediately to security as an operational issue;; 2 = report;; 1 = ignore;; 0 = none.
- EX019 — Pilot agreement. How many of 40 pilot accounts did reviewers agree on? Rubric: 3 = 36;; 2 = 90%;; 1 = vague;; 0 = none.
- EX020 — Catalogue fix. What rule did Lucas add after the pilot? Rubric: 3 = a rule for shared resources such as mailboxes;; 2 = shared mailboxes;; 1 = vague;; 0 = none.
Synthesis
Observational designs need recorded confounders, approvals and a pre-registered plan.
Continue studying — chapter 03 of 08
← Chapter 02 (draft) · Course chapter index · Chapter 04 (draft) →