Chapter 3 — Study design, ethics and reproducibility

Chapter 3 — Study design, ethics and reproducibility

Lesson map

Lesson 03 published. Map, example, EX015.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["3.1 Designs for an event nobody assigned"]
  N2["3.2 Threats to validity"]
  N3["3.3 Design analysis before data"]
  N4["3.4 Ethics review"]
  N5["3.5 Data agreements and the data management plan"]
  N6["3.6 A reproducible project from day one"]
  N7["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX015"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["3.1 Designs for an event nobody assigned"]
  N2["3.2 Threats to validity"]
  N3["3.3 Design analysis before data"]
  N4["3.4 Ethics review"]
  N5["3.5 Data agreements and the data management plan"]
  N6["3.6 A reproducible project from day one"]
  N7["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX015"]

[Learning outcomes and connection]
    |
    v
[3.1 Designs for an event nobody assigned]
    |
    v
[3.2 Threats to validity]
    |
    v
[3.3 Design analysis before data]
    |
    v
[3.4 Ethics review]
    |
    v
[3.5 Data agreements and the data management plan]
    |
    v
[3.6 A reproducible project from day one]
    |
    v
[Chapter synthesis]
    |
    v
[Worked example] --> [Exercise EX015]

Learning outcomes and connection

By the end of this chapter the learner will be able to:

1. Compare the designs available for a causal question about an unplanned infrastructure event — before-and-after, interrupted time series, controlled interrupted time series, difference-in-differences, synthetic control, case study and mixed methods — and choose one by the assumptions it needs and the data it requires.
2. Name the main threats to internal, construct, statistical-conclusion and external validity, and attach each to a design feature that reduces it.
3. Run a design analysis before data access: estimate how large an effect the planned design can detect and how biased a simpler design would be.
4. Prepare an ethics application for a study that combines aggregate operational data and interviews, including risks to staff, organizations and the public, and information-sensitivity safeguards.
5. Specify data agreements and a data management plan proportionate to the data's sensitivity.
6. Set up a reproducible project — structure, environment, seeds, codebook, version control and a registered pre-analysis plan — before the first data arrive.

The chapter turns Chapter 1's estimand into a design. Chapter 4 examines the rival explanations that the design must defeat; Chapter 5 tests feasibility; Chapter 6 executes the analysis specified here.

3.1 Designs for an event nobody assigned

3.1.1 The core problem

A ransomware attack is not an experiment: nobody randomly assigned it to Kestrel or chose its timing to help researchers. Every design for the Spillover Study is therefore a way of approximating the counterfactual — what Alder Point's arrivals would have been without the diversion — from data that exist. Designs differ in where they find that approximation and in the assumptions that make it valid.

If the study pools several events at different times — for example, attacks on different hospital systems in different years — standard two-way fixed-effects difference-in-differences can give misleading weights to some comparisons; Goodman-Bacon (2021) decomposes the estimator and Roth et al. (2023) survey the alternatives. A single-event master's study avoids that problem but gives up generality.

3.1.2 The Spillover Study's choice

The study uses a controlled interrupted time series for RQ1 and RQ2: daily ambulance arrivals at each neighbouring ED over 180 days before and 90 days after day 0, relative to the pooled Wexmoor EDs, with a linear trend, a step for the 21-day diversion window and a step for the following period. It adds interviews for RQ3. The reasons, recorded in the proposal: the comparison region removes shared seasonal change (an uncontrolled ITS cannot); two Wexmoor EDs are too few donors for a synthetic control; daily data give enough time points for the trend; and planners need the mechanism (RQ3) as much as the number.

A methodological review of ITS studies evaluating health-system interventions found that 81% used no control series, only 55% considered autocorrelation, 20.8% seasonality and 8.3% non-stationarity, and only 49.2% specified the expected form of the impact (Hategeka et al., 2020). Those are the common failures this chapter's design is built to avoid.

3.2 Threats to validity

The following vocabulary is standard in quasi-experimental methods. Attach each threat to the design feature that addresses it — and admit the threats no feature addresses.

3.3 Design analysis before data

3.3.1 Why calculate before collecting

For a planned experiment, a sample-size calculation fixes how many participants to recruit. An observational study of one event cannot recruit more events, but it can and should ask before data access: how large an effect could this design detect, and how wrong would a simpler design be? Lakens (2022) lists six ways to justify a sample size, including measuring (almost) the whole population, accepting resource constraints and planning for a desired accuracy; whichever is used, the researcher should state which effects the data can and cannot inform. For the Spillover Study, the sample is fixed by the event: one diversion of 21 days and as many pre-event days as the partners can supply.

3.3.2 The smallest effect of interest

A design analysis needs a reference effect. The Tamsin Valley coalition (synthetic) activates mutual aid when an ED expects four or more extra ambulance arrivals per day — about 10% at Alder Point's baseline of 40. That gives the study a smallest effect of interest grounded in a real decision: 10%. Write where the number came from; an effect size invented to make a study look adequately powered is a common and easily detected weakness.

3.3.3 Simulating the design

The lab below simulates the planned study many times under known truths, using synthetic daily arrivals with day-of-week variation and extra day-to-day variation. It compares an uncontrolled design (the treated ED's change from its own baseline) with a controlled design (the same change minus the change at an unaffected comparison ED), for baselines of 28, 90 and 180 days, with and without a shared seasonal rise starting near day 0. It reports the mean estimated effect (bias) and the share of simulated studies in which the 95% interval excludes zero ("flagged").

# Lab 3 - design check before data access: bias and detection of two designs (offline, standard library only)
# Synthetic daily ambulance arrivals at a neighbouring hospital (treated) and an unaffected comparison hospital.
# Days -pre..-1 are baseline; days 0..20 are the 21-day diversion window at the attacked hospital.
import math, random

DOW = [1.08, 1.02, 1.00, 0.98, 1.00, 0.96, 0.96]

def poisson(rng, lam):                        # Knuth's method; adequate for lam < ~500
    L, k, p = math.exp(-lam), 0, 1.0
    while True:
        p *= rng.random()
        if p <= L:
            return k
        k += 1

def season(t, ramp):                          # respiratory season starting around day 0, shared by both
    return 1 + (0.18 / (1 + math.exp(-(t - 10) / 6)) if ramp else 0.0)

def day(rng, base, t, effect, ramp):
    lam = base * DOW[t % 7] * season(t, ramp) * (1 + effect) * rng.gammavariate(60, 1 / 60)
    return poisson(rng, lam)

def mean_var(x):
    m = sum(x) / len(x)
    return m, sum((v - m) ** 2 for v in x) / (len(x) - 1)

def log_ratio(pre, ev):                       # log(mean_event / mean_pre) and its approximate SE
    mp, vp = mean_var(pre)
    me, ve = mean_var(ev)
    return math.log(me / mp), math.sqrt(ve / (len(ev) * me ** 2) + vp / (len(pre) * mp ** 2))

def one_study(rng, pre_len, effect, ramp):
    days = range(-pre_len, 21)
    tr = {t: day(rng, 40, t, effect if t >= 0 else 0.0, ramp) for t in days}
    co = {t: day(rng, 35, t, 0.0, ramp) for t in days}
    pre, ev = [t for t in days if t < 0], [t for t in days if t >= 0]
    u, su = log_ratio([tr[t] for t in pre], [tr[t] for t in ev])           # uncontrolled pre/post
    c1, s1 = log_ratio([co[t] for t in pre], [co[t] for t in ev])
    return (u, su), (u - c1, math.sqrt(su ** 2 + s1 ** 2))                  # controlled (DiD on logs)

def run(pre_len, effect, ramp, sims=400, seed=11):
    rng = random.Random(seed)
    res = {"uncontrolled": [], "controlled": []}
    for _ in range(sims):
        a, b = one_study(rng, pre_len, effect, ramp)
        res["uncontrolled"].append(a)
        res["controlled"].append(b)
    out = {}
    for k, v in res.items():
        est = [math.exp(e) - 1 for e, _ in v]
        hits = sum(abs(e) / s > 1.96 for e, s in v) / sims
        out[k] = (sum(est) / sims, hits)
    return out

if __name__ == "__main__":
    print("season  true  pre_days  design        mean_est  share_flagged")
    for ramp in (False, True):
        for effect in (0.0, 0.10):
            for pre_len in (28, 90, 180):
                r = run(pre_len, effect, ramp)
                for k in ("uncontrolled", "controlled"):
                    m, h = r[k]
                    print(f"{'ramp' if ramp else 'flat':6}  {effect:4.2f}  {pre_len:8d}  {k:12}  {m:+8.3f}  {h:13.2f}")

Expected output (about three seconds):

season  true  pre_days  design        mean_est  share_flagged
flat    0.00        28  uncontrolled    +0.004           0.06
flat    0.00        28  controlled      +0.010           0.04
flat    0.00        90  uncontrolled    +0.001           0.04
flat    0.00        90  controlled      +0.004           0.02
flat    0.00       180  uncontrolled    +0.004           0.05
flat    0.00       180  controlled      +0.000           0.04
flat    0.10        28  uncontrolled    +0.102           0.39
flat    0.10        28  controlled      +0.115           0.24
flat    0.10        90  uncontrolled    +0.105           0.52
flat    0.10        90  controlled      +0.110           0.33
flat    0.10       180  uncontrolled    +0.101           0.52
flat    0.10       180  controlled      +0.109           0.29
ramp    0.00        28  uncontrolled    +0.084           0.27
ramp    0.00        28  controlled      +0.003           0.07
ramp    0.00        90  uncontrolled    +0.088           0.40
ramp    0.00        90  controlled      +0.002           0.06
ramp    0.00       180  uncontrolled    +0.096           0.52
ramp    0.00       180  controlled      +0.006           0.07
ramp    0.10        28  uncontrolled    +0.191           0.84
ramp    0.10        28  controlled      +0.105           0.23
ramp    0.10        90  uncontrolled    +0.198           0.94
ramp    0.10        90  controlled      +0.106           0.28
ramp    0.10       180  uncontrolled    +0.202           0.97
ramp    0.10       180  controlled      +0.106           0.31

Four conclusions follow, and the proposal must state them before any data are seen:

1. Without a seasonal shift, both designs are unbiased, and the uncontrolled design flags a true 10% effect more often (about 52% versus about 30% with 90 or 180 baseline days) because subtracting a comparison series adds its noise.
2. With a shared seasonal rise, the uncontrolled design is badly biased. When the true effect is zero it estimates about +8% to +10% and flags a non-existent effect in 27% to 52% of studies — more often with longer baselines, because the baseline then reaches further back into the low season. More data did not fix a design problem; it made the false conclusion more confident.
3. The controlled design removes the seasonal bias (mean estimates near zero; flagged in 6–7% of null studies, close to the nominal 5% given the approximate standard errors) but detects a true 10% effect in only 23–31% of studies.
4. Therefore one neighbouring ED cannot reliably detect the smallest effect of interest. The study should be framed as estimation with intervals, not as a test that "shows" spillover; it should report the interval and the effects it rules out; and RQ2's distance gradient should be read across all three neighbours, not from one test. (The controlled estimates average about 0.105–0.115 when the truth is 0.10 because the average of exponentiated log ratios is slightly larger than the exponentiated average — a small, known bias.)

3.4 Ethics review

3.4.1 What needs review

Interviews with ED managers, EMS supervisors and coalition staff are research with human participants in almost every jurisdiction and need approval from the institution's research ethics committee or institutional review board before recruitment. In the United States, the Common Rule (45 CFR Part 46) defines human-subjects research and the categories that may be exempt; the institution, not the student, decides whether a study qualifies. Aggregate daily counts per ED usually contain no information about individuals, but the institution may still require review, and a data custodian may require its own approval. In the EU, health data are a special category under Article 9 of the GDPR, and processing for scientific research relies on safeguards under Article 89. If individual-level records were ever needed, U.S. health data would bring the HIPAA Privacy Rule's de-identification standards and, for a limited data set, a data use agreement (HHS guidance). The Spillover Study avoids individual-level records entirely — data minimization is the strongest ethical safeguard available.

3.4.2 Risks particular to infrastructure studies

The Menlo Report principles — respect for persons, beneficence, justice, and respect for law and public interest — structure the risk analysis. In this study the main risks are:

• To interviewees: being blamed for decisions during the incident; disclosure of identity through role plus organization (there is one EMS operations supervisor in Tamsin Valley); professional consequences of criticizing an employer or a neighbouring hospital.
• To organizations: reputational harm ("Alder Point could not cope"); regulatory exposure; disclosure of security weaknesses at Kestrel.
• To the public: a published finding such as "this region's EDs lose a quarter of their spare capacity when one system diverts" could, combined with other information, help an adversary choose a target that maximizes regional harm. The doctoral book's information-sensitivity test applies: does an artifact identify a specific asset, its vulnerability or its dependency in a way that would help someone harm it?
• To the researcher: pressure from partners to soften results; access to information whose handling the researcher is not trained for.

3.4.3 Safeguards

3.4.4 The application

A typical application contains: the question and its benefit; the design and data sources; recruitment, consent and withdrawal; risks and safeguards; data handling, storage, retention and destruction; dissemination; conflicts of interest; and the researcher's training. Attach the interview guide, information sheet, consent form and data agreements. Write for a committee member who is not a security specialist: explain why a finding about regional capacity is sensitive in plain language.

3.5 Data agreements and the data management plan

Before any data move, put in writing: what data, at what level of aggregation, for which period; purpose limitation; storage location and access; retention and destruction; publication review (factual and protected-information review only, within a fixed number of days, with no veto); acknowledgement and authorship; and breach notification. A data management plan then records, for each data asset, its source, sensitivity tier, format, location, who may access it, retention and what may be shared at the end. Use sensitivity tiers consistently across the project:

3.6 A reproducible project from day one

Reproducibility is easier to build than to retrofit. Sandve et al. (2013) give ten rules, among them: record how every result was produced; avoid manual data manipulation; archive exact versions of external programs; version-control custom scripts; record intermediate results; and note the seeds of random processes. The Turing Way community handbook expands these into practice. The FAIR principles — findable, accessible, interoperable, reusable — apply to what can be shared; for restricted data, FAIR metadata can describe a dataset that cannot itself be released.

A minimal structure:

spillover-study/
  README.md            question, how to run everything, environment, contacts
  data/raw/            read-only copies as received (restricted; never in version control)
  data/derived/        produced only by scripts
  code/01_audit.py     numbered in execution order
  code/02_build.py
  code/03_analysis.py
  output/              tables and figures, regenerated by scripts
  docs/codebook.md     every variable: source, definition, unit, missing-value code
  docs/deviations.md   every change to the plan, dated, with reason and whether results had been seen
  docs/environment.txt Python version and package versions

A pre-analysis plan fixes before outcome data are examined: the estimands, primary and secondary outcomes, comparison units, windows, model, interval method, robustness checks and how deviations will be reported. Registering it with a timestamp — for example in the OSF Registries — makes the plan verifiable; Nosek et al. (2018) describe why preregistration distinguishes prediction from postdiction. In an observational study the data may already exist, so state honestly what had been seen when the plan was registered (for example, "pre-event data for one ED were examined during the pilot for completeness only").

3.7 Worked example: the ethics committee's questions (synthetic)

The committee returned the candidate's first application with five questions. The candidate's responses:

The application was approved with one condition: the consent form had to state that the researcher is not employed by any participating organization and will not share identifiable data with them.

3.8 Common errors and limits

• Choosing a before-and-after design because the data are easy, then claiming a causal effect.
• Using an uncontrolled ITS when a shared shock (season, epidemic, policy) is plausible.
• Assuming parallel trends without inspecting pre-event paths.
• Running a power calculation with an effect size chosen to make the study look adequate, rather than a smallest effect of interest tied to a decision.
• Believing that more baseline data always reduce bias; with a trend or seasonal shift, they can increase it.
• Submitting an ethics application that treats aggregate infrastructure data as risk-free.
• Agreeing to a partner review without a time limit and a "no veto on findings" clause.
• Setting up reproducibility at the end; manual edits to data files that no script records.
• Registering a pre-analysis plan without stating what data had already been seen.

The limit of design is that it can only reduce, not remove, the dependence on untestable assumptions. A controlled ITS is credible only if the comparison region had no different shock at the same time; Chapter 4 tests that assumption as far as data allow.

Exercises — chapter 3

EX015 — Choose and defend a design. Objective: select a design by assumptions and data. Inputs: the designs in §3.1.1; a scenario in which a regional payment processor suffered a three-day outage and you have weekly card-transaction counts for 40 merchant categories in the affected region and in three unaffected regions for two years. Constraints: one primary design and one robustness design. Steps: (1) state the estimand; (2) compare at least three designs on assumption, data and weakness; (3) choose; (4) say what would make you abandon the choice. Expected evidence: one page. Rubric: 3 = weekly resolution's consequence for a three-day event recognized, parallel-trends or donor-pool assumptions stated and checkable, abandonment condition concrete; 2 = one design misdescribed; 1 = no assumptions; 0 = none.

EX016 — Design analysis. Objective: find the smallest spillover effect a controlled before-after design can reliably detect, and reframe RQ1 to match that power. Inputs: Lab 3 code. Constraints: offline; keep seed 11 and 400 simulations. Steps: (1) run the lab and record the output; (2) add a true effect of 0.20 and report the controlled design's flagged share for 28, 90 and 180 baseline days under the seasonal rise; (3) state the smallest effect the controlled design detects in at least about three quarters of studies with 180 days; (4) rewrite RQ1 as an estimation question that this design can honestly answer. Expected evidence: outputs, a sentence on detectable effects, revised RQ1. Rubric: 3 = correct values, detectable effect stated as about 20% rather than 10%, RQ1 reframed toward estimation; 2 = one value wrong; 1 = not run; 0 = none.

EX017 — Ethics application risks. Objective: identify and mitigate risks. Inputs: a study interviewing water-utility operators about a disinfection-system outage at a small utility. Constraints: use the four Menlo principles; at least five stakeholders. Steps: (1) list stakeholders and risks; (2) propose safeguards; (3) identify the information-sensitivity risk specific to a small utility; (4) draft the two sentences of the information sheet about confidentiality limits. Expected evidence: table and draft text. Rubric: 3 = small-population identification risk recognized, public-targeting risk addressed, safeguards concrete, information-sheet text honest about limits; 2 = one principle thin; 1 = generic; 0 = none.

EX018 — Data management plan. Objective: tier and handle data. Inputs: the Spillover Study's assets: custodian daily counts; Wexmoor counts from a public dashboard; interview recordings; transcripts; codebook; analysis code; consent forms; the coalition's mutual-aid thresholds. Constraints: the three tiers in §3.5. Steps: assign a tier, storage, access, retention and end-of-project sharing for each. Expected evidence: table. Rubric: 3 = tiers consistent with sensitivity (recordings and consent forms confidential, custodian counts restricted, code and codebook public), public-dashboard data handled according to its terms, mutual-aid thresholds treated as sensitive; 2 = one misassigned; 1 = everything public or everything confidential; 0 = none.

EX019 — Pre-analysis plan. Objective: fix the analysis before outcomes are seen. Inputs: §3.1.2 and Chapter 1's estimands. Constraints: two pages at most. Steps: specify estimands, outcomes (primary and secondary), comparison, windows, model, interval method, robustness checks, multiplicity handling and the deviation rule; state what data had been seen at registration. Expected evidence: plan. Rubric: 3 = every element specific enough that another analyst would produce the same estimate, deviations rule explicit, prior data access disclosed; 2 = one element vague; 1 = model unspecified; 0 = none.

EX020 — Reproducibility audit. Objective: test whether a project can be rerun. Inputs: a folder containing the Chapter 1 and 3 labs and their outputs. Constraints: a second person reruns it from the README alone. Steps: (1) write the README; (2) record the environment; (3) have someone else rerun it; (4) list every point where they had to ask you something and fix it. Expected evidence: README, environment file and an issues list with fixes. Rubric: 3 = rerun succeeded with identical outputs and every question answered in the README; 2 = one manual step remains; 1 = outputs differ unexplained; 0 = none.

Chapter synthesis

An unplanned infrastructure event offers no randomization, so every design is an approximation to a counterfactual, and its credibility rests on assumptions you must state. A controlled interrupted time series removes shared seasonal shocks that a before-and-after or uncontrolled ITS would attribute to the event; interviews supply the mechanism. Attach every validity threat to a design feature, or admit it. Simulate the design before data access: Lab 3 shows that a shared seasonal rise makes an uncontrolled design confidently wrong — more so with longer baselines — while the controlled design is nearly unbiased but too imprecise to detect a 10% effect at one ED. Frame the study as estimation. Ethics review must cover interviewees, organizations and the public, because regional-capacity findings can be sensitive. Agreements, tiers and a reproducible project with a registered pre-analysis plan come before the first data file.

Continue studying — chapter 03 of 08

← Chapter 02 · Course chapter index · Chapter 04 →

Continue to the next authored lesson.