◇ OpenCyber Academy
Master's · Chapter 3 · English

Chapter 3 — Study design, ethics and reproducibility

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 3 — Study design, ethics and reproducibility

Lesson map

Lesson 03 published. Map, example, EX015.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["3.1 Designs for incident-response questions"]
  N2["3.2 Threats to validity"]
  N3["3.3 Design analysis before data"]
  N4["3.4 Ethics with analysts as participants"]
  N5["3.5 Predeclared measures and the analysis plan"]
  N6["3.6 A reproducible project from day one"]
  N7["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX015"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["3.1 Designs for incident-response questions"]
  N2["3.2 Threats to validity"]
  N3["3.3 Design analysis before data"]
  N4["3.4 Ethics with analysts as participants"]
  N5["3.5 Predeclared measures and the analysis plan"]
  N6["3.6 A reproducible project from day one"]
  N7["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX015"]

[Learning outcomes and connection]
    |
    v
[3.1 Designs for incident-response questions]
    |
    v
[3.2 Threats to validity]
    |
    v
[3.3 Design analysis before data]
    |
    v
[3.4 Ethics with analysts as participants]
    |
    v
[3.5 Predeclared measures and the analysis plan]
    |
    v
[3.6 A reproducible project from day one]
    |
    v
[Chapter synthesis]
    |
    v
[Worked example] --> [Exercise EX015]

Learning outcomes and connection

After this chapter you should be able to:

  1. Compare observational, quasi-experimental and controlled tabletop designs for an incident-response question and justify a choice in terms of the counterfactual each provides.
  2. Explain the logic of a crossover design, its assumptions (no lasting carryover, balanced periods and case sets) and how to check them.
  3. Size a study by simulation — design analysis — reporting power, the exaggeration of significant estimates and the false-positive rate, rather than quoting a single power formula.
  4. Write an ethics protocol for research with analysts as participants, including consent, voluntariness, confidentiality from employers and handling of security-sensitive material.
  5. Predeclare measures and an analysis plan, and set up a reproducible project before any data exist.

The chapter builds on the question and estimand of Chapter 1 and the outcome choices of Chapter 2, and prepares Chapter 4, which examines the explanations a design must rule out.

3.1 Designs for incident-response questions

3.1.1 The core problem

A causal question asks what would have happened under a different condition — the counterfactual. We never observe it directly: an analyst either used the rubric on a case or did not. Every design is a strategy for approximating the missing counterfactual, and each strategy rests on assumptions you must state and, where possible, test. A standard introduction to this way of thinking is Hernán and Robins, Causal Inference: What If, freely available from its authors.

3.1.2 Design options

Design How the counterfactual is approximated Main assumption Fit for the Triage Rubric Study
Cross-sectional comparison of teams or tickets Another team's tickets in the same period The teams are comparable apart from the rubric Poor: East and West differ in seniority and case mix (Chapter 4).
Before–after (pre–post) in one team The same team before adoption Nothing else changed at the same time Poor: seniority changed in the same month.
Difference-in-differences on historical tickets The other team's change over the same period Parallel trends absent the rubric Possible as a secondary strand; vulnerable to team-specific co-interventions.
Parallel controlled tabletop Other analysts, randomly assigned to usual judgement Randomization balances analysts Valid but needs many analysts, because analysts differ a lot.
Crossover controlled tabletop The same analyst on comparable cases without the rubric No lasting carryover; periods and case sets balanced Chosen: removes stable between-analyst differences; feasible with a few dozen volunteers.

In a crossover design each participant experiences both conditions in sequence. The comparison of interest is within each participant, so stable differences between people — skill, caution, speed — cancel out. The price is three assumptions. (1) No lasting carryover: the first condition must not change behaviour in the second. A rubric is a worrying case, because analysts who used it first may keep its questions in mind afterwards. The usual remedy is a washout or, here, an order that can be checked: if rubric-first analysts do better in their later "usual" period than usual-first analysts do in their earlier one, carryover is suspected. (2) No unbalanced period effect: practice or fatigue may change performance from the first to the second period; balancing the order (half start with each condition) prevents this from biasing the comparison, and it can be estimated. (3) Comparable case sets: each period uses different cases, so the sets must be matched in reference severity and type and counterbalanced across sequences.

The Triage Rubric Study therefore uses four sequences, six analysts each: usual on set A then rubric on set B; usual on B then rubric on A; rubric on A then usual on B; rubric on B then usual on A. Each set has twelve fictional cases with the same distribution of reference severities: eight high-impact (four "quiet"), two of severity 2 and two of severity 1. Analysts are allocated to sequences at random, stratified by experience (more or less than three years), using a seeded random number generator whose seed is recorded in the protocol.

3.1.3 The tabletop as an instrument

A controlled tabletop is a measurement instrument and must be designed like one. NIST SP 800-84 (2006) provides vocabulary for exercise design — objectives, scenarios, injects, facilitators, evaluators — that is still useful, although it was written for organizational exercises rather than experiments. For research use, add three properties. Standardization: every analyst sees the same case materials in the same format, with the same time limit and instructions. A reference standard set independently: three senior responders not involved in the study rate each case from its full fictional ground truth, independently, and then reach consensus; disagreements are recorded (two cases, C07 and C19, are split 2–1, which Chapter 8 uses). Fidelity checks: an observer records whether the rubric sheet was actually completed in rubric sessions and was absent in usual sessions.

3.2 Threats to validity

Validity vocabulary is a standard methods concept; four families are enough for planning.

For each threat, the protocol records a mitigation and a check. For example: observers are not told the hypothesis, and timing is checked by two observers on a sample (Chapter 5); clustering is handled by analysing per-analyst differences (Chapter 6); external validity is discussed with direction, not hidden (Chapter 7).

3.3 Design analysis before data

3.3.1 Why simulate

Classical power formulas answer one question — the probability of a "significant" result if a specified effect is true — under simplifying assumptions. For a crossover with binary outcomes, clustered decisions and analysts who differ, the assumptions are easier to get wrong than to get right. A design analysis by simulation asks a broader set of questions (Gelman and Carlin 2014): if the true effect is what we think, how often will the study detect it (power)? When it does, how much will the estimate exaggerate the truth (type M, magnitude error)? How often will it get the sign wrong (type S)? And if there is no effect, how often will it falsely find one? A study with low power is not merely likely to miss a real effect; its significant results are systematically too large.

3.3.2 The smallest effect of interest

Before simulating, decide the smallest effect of interest: the effect below which the decision-maker would not change practice. MRSS managers say they would not mandate a rubric that costs time unless it cut under-triage of high-impact cases by at least 10 percentage points. That number, not the effect you hope for, should anchor the design (Lakens 2022 reviews ways of justifying sample sizes, including this one). The study also simulates a plausible effect of 15 points, taken from the synthetic pilot literature in the concept matrix.

3.3.3 Simulating the design

Lab 3 simulates thousands of synthetic studies. Each analyst has a personal tendency to under-triage (a random effect on the log-odds scale), rates a fixed number of high-impact cases per condition, and is either in a crossover (both conditions) or a parallel design (one condition, twice as many cases, so the workload is equal). Save it as lab3_design.py.

# Lab 3 - design analysis before recruiting: crossover versus parallel tabletop design (offline, stdlib only)
# Synthetic analysts differ in their own tendency to under-triage. Each analyst rates HI high-impact cases per
# condition. Outcome: under-triage proportion. We simulate many studies to see what each design can detect.
import math, random, statistics as st
def logit_inv(x): return 1 / (1 + math.exp(-x))
def one_study(rng, n_analysts, hi_per_condition, p_usual, p_rubric, design, sd_analyst=0.8):
    base = math.log(p_usual / (1 - p_usual)); eff = math.log(p_rubric / (1 - p_rubric)) - base
    diffs, rub, usu = [], [], []
    for a in range(n_analysts):
        u = rng.gauss(0, sd_analyst)
        rate = lambda cond: sum(rng.random() < logit_inv(base + u + (eff if cond else 0)) for _ in range(hi_per_condition)) / hi_per_condition
        if design == "crossover":
            diffs.append(rate(True) - rate(False))
        else:   # parallel: each analyst sees one condition, with twice the cases so total workload is equal
            hi_per_condition *= 2
            (rub if a % 2 == 0 else usu).append(rate(a % 2 == 0))
            hi_per_condition //= 2
    if design == "crossover":
        est = st.mean(diffs); se = st.stdev(diffs) / math.sqrt(len(diffs))
    else:
        est = st.mean(rub) - st.mean(usu); se = math.sqrt(st.variance(rub) / len(rub) + st.variance(usu) / len(usu))
    return est, abs(est / se) > 2.07 if se > 0 else False   # t(0.975, ~22 df) ~ 2.07
def design_check(design, n, hi, p_u, p_r, reps=2000, seed=7):
    rng = random.Random(seed)
    res = [one_study(rng, n, hi, p_u, p_r, design) for _ in range(reps)]
    sig = [e for e, s in res if s]
    true = p_r - p_u
    power = len(sig) / reps
    exagg = st.mean(abs(e) for e in sig) / abs(true) if sig and true else float("nan")
    wrong_sign = sum(1 for e in sig if e * true < 0) / len(sig) if sig and true else float("nan")
    return power, exagg, wrong_sign, st.mean(e for e, _ in res)
print("design     analysts  hi/cond  true_diff  power  exaggeration_if_sig  mean_estimate")
for design in ("crossover", "parallel"):
    for n in (12, 24, 36):
        p, x, w, m = design_check(design, n, 4, 0.30, 0.15)
        print(f"{design:<10} {n:>8}  {4:>7}  {-0.15:>9.2f}  {p:>5.2f}  {x:>19.2f}  {m:>13.3f}")
p, x, w, m = design_check("crossover", 24, 4, 0.30, 0.30)
print(f"null check (no true effect), crossover n=24: false-positive rate {p:.3f}, mean estimate {m:.3f}")
p, x, w, m = design_check("crossover", 24, 4, 0.30, 0.20)
print(f"smallest effect of interest (-0.10), crossover n=24: power {p:.2f}, exaggeration if significant {x:.2f}")
for hi in (8,):
    p, x, w, m = design_check("crossover", 24, hi, 0.30, 0.15)
    print(f"revised design, crossover n=24 with {hi} high-impact cases per condition: power {p:.2f}, exaggeration {x:.2f}")
    p, x, w, m = design_check("crossover", 24, hi, 0.30, 0.20)
    print(f"revised design at the smallest effect of interest (-0.10): power {p:.2f}, exaggeration {x:.2f}")

Expected output:

design     analysts  hi/cond  true_diff  power  exaggeration_if_sig  mean_estimate
crossover        12        4      -0.15   0.37                 1.47         -0.142
crossover        24        4      -0.15   0.64                 1.16         -0.142
crossover        36        4      -0.15   0.81                 1.07         -0.144
parallel         12        4      -0.15   0.26                 1.81         -0.146
parallel         24        4      -0.15   0.43                 1.45         -0.146
parallel         36        4      -0.15   0.56                 1.27         -0.146
null check (no true effect), crossover n=24: false-positive rate 0.046, mean estimate 0.003
smallest effect of interest (-0.10), crossover n=24: power 0.29, exaggeration if significant 1.60
revised design, crossover n=24 with 8 high-impact cases per condition: power 0.92, exaggeration 1.02
revised design at the smallest effect of interest (-0.10): power 0.56, exaggeration 1.24

How to read it. With four high-impact cases per condition — the first draft protocol — a crossover with 24 analysts detects a 15-point effect only about 64% of the time, and a parallel design of the same size only 43%: between-analyst variation is large, and the crossover removes it. At 12 analysts, significant crossover estimates exaggerate the true effect by about half. The null check shows the analysis keeps its false-positive rate near the nominal 5%. The revision that matters is the number of high-impact cases per condition: with eight, the crossover with 24 analysts has power of about 0.92 for a 15-point effect, with almost no exaggeration. At the smallest effect of interest (10 points), power is still only about 0.56. The protocol says so honestly: the study is designed to detect a plausible effect and to estimate it with useful precision, but a true 10-point effect has roughly even odds of reaching conventional significance, and the dissertation will report an interval, not a verdict.

Two details are worth noticing. The "true difference" is set on each analyst's log-odds scale, so the average difference in proportions across heterogeneous analysts (the mean estimate, about −0.14) is slightly smaller than −0.15 — the estimand from Chapter 1 is the average difference in proportions, and the simulation shows the two are not identical. And the critical value 2.07 approximates a t distribution with about 22 degrees of freedom; Chapter 6 uses resampling methods that avoid this approximation.

3.4 Ethics with analysts as participants

3.4.1 What needs review

Research that collects data from people — their decisions, times, words — normally needs approval from the university's research ethics committee (in the United States an Institutional Review Board under the Common Rule, 45 CFR 46). Using fictional cases does not remove the need: the participants are real, and their performance data are personal data. The Belmont Report principles of respect for persons, beneficence and justice underpin most committees' reasoning, and the Menlo Report (2012) adapts them to information and communication technology research, adding respect for law and public interest. Where participants are in the European Union, the General Data Protection Regulation applies to their personal data. Regulations differ by country and change; the protocol states which apply and the committee decides. This book gives no legal advice.

3.4.2 Risks particular to studying analysts

3.4.3 Safeguards

Informed consent in writing, with a plain-language information sheet; the right to withdraw, including withdrawal of data up to a stated date; pseudonymous participant codes (A01–A24) generated by the student, with the key stored separately and destroyed after the study; no individual results shared with any employer; aggregated reporting with minimum cell sizes; fictional case material reviewed by a second person for real-world leakage; a debrief that explains the study's purpose after all sessions are complete. The debrief interviews (RQ3) have their own consent item, and recordings are deleted after transcription and checking.

3.4.4 The application

An ethics application for this study answers, at minimum: what will participants do and for how long; how they are recruited and by whom; what data are collected, where they are stored, who can access them and when they are deleted; what risks exist and how each is reduced; and how results are reported. Attach the information sheet, consent form, case materials (or a sample), debrief guide and data management plan.

3.5 Predeclared measures and the analysis plan

A pre-analysis plan fixes, before data collection, the hypotheses, primary and secondary outcomes, the estimand, the analysis method, the handling of missing data and deviations, and the robustness checks. It can be registered publicly with a time stamp; preregistration's value is to separate confirmatory from exploratory analysis, not to forbid exploration (Nosek et al. 2018). The case for doing so in fields with many analytic choices was made forcefully by Simmons, Nelson and Simonsohn (2011), who showed how undisclosed flexibility in data collection and analysis can make false findings easy to produce.

The Triage Rubric Study's plan includes:

3.6 A reproducible project from day one

Reproducibility means another person can regenerate every number in the dissertation from the archived data and code. It is cheap at the start and expensive at the end (Stodden et al. 2016 set out practical recommendations for computational work). Set up the project before the pilot:

triage-rubric-study/
  README.md            what the project is, how to run everything, software versions
  protocol/            ethics approval (redacted), pre-analysis plan, deviations log
  instruments/         case vignettes, rubric sheet, observer form, debrief guide
  data/raw/            read-only exports as received; never edited
  data/derived/        produced only by code
  code/                one script per step: clean, analyse, figures
  output/              tables and figures produced by code
  notes/               dated research notes with observation / inference / claim / decision

Rules: raw data are never edited by hand; every derived file is produced by a script; scripts record their seed and software version; the dissertation's numbers come from output files, not from copying. Where data cannot be shared because of confidentiality, the code and a synthetic dataset with the same structure can be, which lets others check the pipeline. The FAIR principles — findable, accessible, interoperable, reusable (Wilkinson et al. 2016) — guide what to archive and describe, even when access must be restricted.

3.7 Worked example: the ethics committee's questions (synthetic)

The committee returns the application with four questions. Each answer changes the protocol.

  1. "The student previously worked at MRSS. Could participants feel obliged?" — Recruitment moves to a professional association's mailing list and two other organizations; MRSS analysts may join, but no MRSS manager is told who did. The student does not recruit former colleagues directly.
  2. "Could results be traced to individuals?" — Tables will report no cell combining team, experience band and sequence; individual analyst estimates appear only as an anonymous dot plot without labels.
  3. "Do the cases contain anything harmful?" — A second reviewer checks every vignette for real indicators and real organization names; the cases use fictional services and documentation ranges for addresses.
  4. "What happens if a participant becomes distressed?" — The facilitator pauses the session; the participant may stop without explanation and keep any thank-you voucher; the information sheet lists support contacts.

The revised protocol is resubmitted with a table mapping each question to the change made.

3.8 Common errors and limits

The limit of design is that every design rests on assumptions. Good design makes them explicit, checkable and few.

Exercises — chapter 3

EX015 — Choose a design. Objective: justify a design by its counterfactual. Inputs: the design table in §3.1.2. Constraints: one primary design; at most one secondary strand. Steps: (1) for each of the five designs, state its counterfactual and main assumption for RQ1; (2) choose one and justify it in 150 words; (3) state what the chosen design cannot answer and which strand, if any, will address it. Expected evidence: a table and paragraph. Rubric: 3 = counterfactuals precise, choice justified by assumptions and feasibility, limits of the chosen design acknowledged; 2 = one design misdescribed; 1 = choice by label; 0 = none.

EX016 — Carryover and period checks. Objective: design the checks for a crossover. Inputs: the four sequences in §3.1.2. Constraints: use only data the study collects. Steps: (1) define the period effect and show how the balanced sequences let you estimate it; (2) define a carryover check; (3) say what you would do if carryover were found. Expected evidence: definitions and a decision rule. Rubric: 3 = period effect estimated from within-analyst changes by order, carryover check compares rubric-first and usual-first analysts' usual periods, response honest (for example, report the first-period comparison as a parallel design with reduced precision); 2 = one check missing; 1 = assumptions named only; 0 = none.

EX017 — Run and extend Lab 3. Objective: use design analysis to size a study. Inputs: Lab 3. Constraints: offline; add lines, do not change the existing ones. Steps: (1) run it and record the output; (2) add a run of the crossover with 36 analysts and 8 high-impact cases per condition at the smallest effect of interest (0.30 versus 0.20); (3) report power and exaggeration; (4) recommend 24 or 36 analysts given that recruitment beyond 24 adds an estimated three months. Expected evidence: outputs and a recommendation. Rubric: 3 = outputs correct, recommendation weighs precision at the smallest effect against time and states what will be reported either way; 2 = recommendation by power alone; 1 = not run; 0 = none.

EX018 — Ethics risk table. Objective: identify and mitigate risks. Inputs: §3.4. Constraints: at least six risks. Steps: for each risk give likelihood, severity, mitigation and residual risk. Expected evidence: a table. Rubric: 3 = risks specific to analysts and security material (employment, coercion, re-identification, leakage, distress, data retention), mitigations concrete, residual risk honest; 2 = generic risks; 1 = fewer than four; 0 = none.

EX019 — Pre-analysis plan. Objective: fix the analysis before data. Inputs: §3.5. Constraints: one page; every outcome defined operationally. Steps: write the plan, including a rule for what counts as a deviation and how deviations are logged. Expected evidence: the plan. Rubric: 3 = outcomes, estimand, method, missing data, checks and robustness all specified, confirmatory and exploratory analyses separated; 2 = one element vague; 1 = a list of intentions; 0 = none.

EX020 — Reproducible project. Objective: set up a project another person could rerun. Inputs: §3.6. Constraints: data cannot be shared publicly. Steps: (1) write the README a stranger would need; (2) describe the synthetic stand-in dataset you will publish and how it preserves structure without disclosing anyone; (3) list what a reproducibility reviewer would need access to under agreement. Expected evidence: README and a paragraph. Rubric: 3 = README runnable step by step with versions and seeds, synthetic stand-in described concretely, restricted access defined; 2 = one gap; 1 = folder list only; 0 = none.

Chapter synthesis

Design is the choice of counterfactual. Historical comparisons are vulnerable to everything that changed alongside the exposure; a controlled crossover tabletop removes stable differences between analysts at the cost of assumptions about carryover, periods and case sets, which must be balanced and checked. Size the study by simulation, reporting power, exaggeration and the false-positive rate at the smallest effect of interest. Treat ethics as design: analysts are participants whose performance data could harm them, and safeguards change recruitment, reporting and storage. Fix outcomes and analysis in a pre-analysis plan, and build a reproducible project before the first datum arrives.

Continue studying — chapter 03 of 08

← Chapter 02 · Course chapter index · Chapter 04 →

Continue to the next authored lesson.

Worked answers

EX015 — Choose a design. Counterfactuals: cross-sectional — West's tickets stand in for East-without-rubric (assumes teams comparable); pre–post — East-2024 stands in for East-2025-without-rubric (assumes nothing else changed); DiD — West's change stands in for East's change without rubric (parallel trends); parallel tabletop — randomized other analysts stand in (randomization balances analysts, but variance is large); crossover — the same analyst on matched cases stands in (no carryover, balanced periods, comparable sets). Choice: crossover tabletop, because it removes stable analyst differences, which Lab 3 shows dominate variance, and because its assumptions can be balanced and checked; recruitment of 24 is feasible. Cannot answer: prevalence of under-triage and effect in live operations; the historical strand provides context on prevalence and on whether the operational data are consistent with an effect, not a causal estimate.

EX016 — Carryover and period checks. Period effect: for each analyst compute (period-2 under-triage − period-1 under-triage). In U→R sequences this equals rubric effect + period effect; in R→U sequences it equals −rubric effect + period effect. Half the sum of the two group means estimates the period effect; half the difference estimates the rubric effect. Carryover: compare the usual-judgement proportions of rubric-first analysts (their period 2) with those of usual-first analysts (their period 1), adjusting for the period effect; carryover predicts better usual-judgement performance among rubric-first analysts. Formal tests of carryover in a two-period crossover have low power, so the check is descriptive. If found: report the period-1 comparison (rubric-first versus usual-first analysts, each in their first period) as a parallel-group estimate free of carryover, with its wider interval, alongside the crossover estimate, and discuss the difference.

EX017 — Run and extend Lab 3. Adding:

p, x, w, m = design_check("crossover", 36, 8, 0.30, 0.20)
print(f"EX017: crossover n=36, 8 high-impact per condition, effect -0.10: power {p:.2f}, exaggeration {x:.2f}")

prints EX017: crossover n=36, 8 high-impact per condition, effect -0.10: power 0.73, exaggeration 1.10. Recommendation: at the smallest effect of interest, 36 analysts raise power from about 0.56 to 0.73 and reduce exaggeration from 1.24 to 1.10. That is a real gain, but three extra months threaten the critical path. A defensible recommendation is to plan for 24 with an extension to 36 if recruitment runs ahead of schedule, and in either case to report the estimate and interval rather than a significance verdict; the dissertation states that a true 10-point effect may not reach conventional significance with 24 analysts.

EX018 — Ethics risk table.

Risk Likelihood Severity Mitigation Residual
Employer sees individual results Low High No individual results shared; pseudonyms; key separate Low
Coercion via manager or ex-colleague recruitment Medium Medium Neutral channels; non-participation invisible Low
Re-identification in small tables Medium Medium Minimum cell sizes; no team × experience × sequence cells Low–medium
Real indicators or configurations in cases Low High Second-person review of vignettes; fictional systems Low
Distress under timed simulation Low Medium Right to stop; pauses; support contacts Low
Interview data retained too long Medium Medium Delete recordings after checking; retention period stated Low
Incidental disclosure of real weaknesses in debrief Low High Instruct participants not to discuss real systems; do not record such content Low

EX019 — Pre-analysis plan. A one-page plan containing: hypotheses (H1: rubric reduces under-triage of high-impact cases; H2: the reduction is larger in quiet-impact cases); primary outcome and estimand as in EX007; analysis — mean of per-analyst differences, percentile bootstrap (4,000 resamples of analysts, seed 1), sign-flip test (20,000 flips, seed 2); secondary outcomes as in EX013; checks — period, carryover (descriptive), fidelity; robustness — two-level threshold, severity ≥ 2 inclusion, critical-only, minority reference for split cases; missing data — complete pairs primary, bounds for withdrawals; deviation rule: any change after the first main-study session is logged with date, reason, whether outcome data had been examined, and whether the change affects confirmatory analyses; changes made after outcome data were examined render affected analyses exploratory. Confirmatory: H1 only; everything else secondary or exploratory.

EX020 — Reproducible project. README (outline): purpose; licence; requirements (Python 3.13, standard library only); folder layout as in §3.6; "to reproduce: (1) place the raw exports in data/raw/ (restricted) or run code/make_synthetic.py to create the synthetic stand-in; (2) run code/clean.py; (3) run code/analyse.py; (4) run code/check_numbers.py against dissertation/numbers.csv"; seeds listed; expected outputs listed with hashes. Synthetic stand-in: generated by the same code as Lab 6 with the same variables, case structure and sequences, but participant-level values drawn from the generator, so it reproduces the pipeline and table shapes without describing any real person. Restricted access: a reviewer may access the de-identified decision-level data in the university's secure environment under a data use agreement; transcripts are not available.