Chapter 6 — Data collection and analysis

Chapter 6 — Data collection and analysis

Lesson map

Lesson 06 published. Map, example, EX034.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["6.1 Collecting data with lineage"]
  N2["6.2 The controlled interrupted time-series model"]
  N3["6.3 Autocorrelation and robust intervals"]
  N4["6.4 Placebo windows as evidence"]
  N5["6.5 Practical activity: the main quantitative analys"]
  N6["6.6 The qualitative strand"]
  N7["6.7 Integration: the joint display"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX034"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["6.1 Collecting data with lineage"]
  N2["6.2 The controlled interrupted time-series model"]
  N3["6.3 Autocorrelation and robust intervals"]
  N4["6.4 Placebo windows as evidence"]
  N5["6.5 Practical activity: the main quantitative analys"]
  N6["6.6 The qualitative strand"]
  N7["6.7 Integration: the joint display"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX034"]

[Learning outcomes and connection]
    |
    v
[6.1 Collecting data with lineage]
    |
    v
[6.2 The controlled interrupted time-series model]
    |
    v
[6.3 Autocorrelation and robust intervals]
    |
    v
[6.4 Placebo windows as evidence]
    |
    v
[6.5 Practical activity: the main quantitative analys]
    |
    v
[6.6 The qualitative strand]
    |
    v
[6.7 Integration: the joint display]
    |
    v
[Worked example] --> [Exercise EX034]

Learning outcomes and connection

By the end of this chapter the learner will be able to:

1. Run a data-collection procedure that preserves lineage: every file received, audited, frozen and traceable to the analysis that used it.
2. Specify and fit a controlled interrupted time-series model by segmented regression, and interpret each coefficient.
3. Diagnose residual autocorrelation and compute autocorrelation-robust (Newey–West) intervals, reporting how the choice of lag affects them.
4. Use placebo windows as rank-based evidence and state the smallest p-value a given number of placebos can produce.
5. Conduct and analyse interviews with a documented thematic analysis, and use coder agreement as a reflexive check rather than a guarantee.
6. Integrate quantitative and qualitative strands in a joint display, naming where they confirm, expand or conflict.

This chapter executes the plan fixed in Chapters 3 and 5 and confronts the rivals of Chapter 4 with data. Chapter 7 writes what it finds; Chapter 8 defends it.

6.1 Collecting data with lineage

Every number in the dissertation must be traceable to a file, a script and a version. A minimal procedure for each extract:

1. Receive through the channel named in the data agreement; never by personal email.
2. Record the file name, sender, date received and a cryptographic checksum (for example SHA-256) in a data log.
3. Store the original read-only in the restricted area (Chapter 3's tiers).
4. Audit with the Lab 5 script; send findings to the custodian; record their answers in the codebook.
5. Freeze: when an extract passes, record its checksum as the analysis version. Derived files are produced only by scripts from frozen inputs.
6. Capture public sources (here, Wexmoor's dashboard) with the date of capture, because dashboards are revised.

The Spillover Study received daily aggregates for days −180 to +89 from three custodians and captured Wexmoor's series from the regional dashboard on a recorded date. All passed the audit after one round of corrections, and custodian-flagged downtime days were recorded in the codebook. (The lab generator does not simulate downtime days, so the labs analyse every day.)

6.2 The controlled interrupted time-series model

For each neighbouring ED, the outcome on day t is the log ratio of its arrivals to the pooled Wexmoor arrivals:

yₜ = log(Aₜ + 0.5) − log(Wₜ + 0.5) = β₀ + β₁·t + β₂·Dₜ + β₃·Pₜ + eₜ

where Dₜ = 1 during the diversion window (days 0–20) and Pₜ = 1 after it (days 21–89). Reading the coefficients:

• β₀, β₁: the pre-event level and linear trend of the ratio. If the treated and comparison regions share the season, the ratio is roughly flat even when each series is not.
• β₂: the change in the log ratio during diversion, relative to the projected trend. exp(β₂) − 1 is the estimated percentage change — the primary estimate.
• β₃: the change after diversion ends, which absorbs any tail so that post-event days do not contaminate the trend.

Choices and their reasons, all fixed in the pre-analysis plan: the log ratio treats effects as multiplicative, which suits counts whose variability grows with their level, and removes shared day-of-week and seasonal patterns; adding 0.5 avoids log(0) on very small counts and has negligible effect at these volumes; a linear trend over 180 days captures slow drift without chasing noise; a separate post-window step avoids assuming that the system returns to its trend the day diversion ends. Lopez Bernal, Cummins and Gasparrini (2017) give a tutorial on segmented regression for ITS, including the need to specify the impact model (a step, a slope change, or both) in advance; the Spillover Study prespecified a step.

6.3 Autocorrelation and robust intervals

Daily operational series are rarely independent from one day to the next: a busy day may be followed by another busy day. Ordinary least-squares coefficients remain unbiased under autocorrelation, but their usual standard errors are wrong — usually too small when autocorrelation is positive. Two responses are common:

1. Diagnose: inspect residual autocorrelation at lags 1 and 7, or compute the Durbin–Watson statistic (Durbin and Watson, 1950); values near 2 suggest little first-order autocorrelation.
2. Make inference robust: use heteroskedasticity- and autocorrelation-consistent (HAC) standard errors such as the Newey–West (1987) estimator, which weights cross-products of residuals up to a chosen lag L with declining Bartlett weights 1 − l/(L + 1). Alternatively, model the error process directly (for example with ARIMA errors), as Lopez Bernal et al. discuss.

The Spillover Study prespecified Newey–West with lag 7 (one week, matching the weekly cycle) and a sensitivity analysis with lags 0 and 14. For Alder Point, the residuals showed lag-1 autocorrelation of 0.03, lag-7 autocorrelation of −0.12 and a Durbin–Watson statistic of 1.92: autocorrelation is weak, so the robust interval should not differ greatly from the naive one — and it does not (EX034).

6.4 Placebo windows as evidence

Chapter 4 introduced placebo-in-time tests. Here they are made precise: fit the same model to fake 21-day windows placed in the baseline, using only data before the real event, and record each fake "effect". If the real estimate is larger than all k placebo estimates, its rank is 1 of k + 1, and under the assumption that the real window is exchangeable with the fake ones, the one-sided probability of a rank that extreme by chance is 1/(k + 1). With 15 placebos the smallest attainable value is 1/16 = 0.0625. Placebo tests therefore cannot reach 0.05 with fewer than 19 windows (or fall below it with fewer than 20), whatever the data; report the rank, not a misleading p-value. The placebo windows also have a weakness: fits near the start of the series rest on short pre-periods and are noisier than the real fit, which makes the test conservative.

6.5 Practical activity: the main quantitative analysis

Save Lab 4 as lab4_data.py and the following file as lab6_its.py in the same folder (Lab 8 imports it). It implements least squares by Gauss–Jordan elimination, Newey–West standard errors and the controlled ITS of §6.2, fits it for each neighbour against pooled Wexmoor, runs 15 placebo windows per site and fits an uncontrolled ITS for Alder Point's waiting time.

# Lab 6 - controlled interrupted time series with autocorrelation-robust intervals (offline)
# Save Lab 4 as lab4_data.py in the same folder first. Save this file as lab6_its.py (Lab 8 imports it).
import math
from lab4_data import make_region, DAYS

def solve(A, b):                                   # Gauss-Jordan with partial pivoting
    n = len(A)
    M = [row[:] + [b[i]] for i, row in enumerate(A)]
    for c in range(n):
        p = max(range(c, n), key=lambda r: abs(M[r][c]))
        M[c], M[p] = M[p], M[c]
        for r in range(n):
            if r != c:
                f = M[r][c] / M[c][c]
                M[r] = [x - f * y for x, y in zip(M[r], M[c])]
    return [M[i][n] / M[i][i] for i in range(n)]

def inverse(A):
    n = len(A)
    cols = [solve(A, [1.0 if i == j else 0.0 for i in range(n)]) for j in range(n)]
    return [[cols[j][i] for j in range(n)] for i in range(n)]

def ols_nw(X, y, lag=7):
    k = len(X[0])
    XtX = [[sum(r[i] * r[j] for r in X) for j in range(k)] for i in range(k)]
    beta = solve(XtX, [sum(r[i] * v for r, v in zip(X, y)) for i in range(k)])
    e = [v - sum(b * x for b, x in zip(beta, r)) for r, v in zip(X, y)]
    S = [[0.0] * k for _ in range(k)]
    for l in range(lag + 1):                         # Newey-West with Bartlett weights
        w = 1.0 if l == 0 else 1 - l / (lag + 1)
        for t in range(l, len(y)):
            for i in range(k):
                for j in range(k):
                    v = e[t] * e[t - l] * X[t][i] * X[t - l][j]
                    S[i][j] += w * (v if l == 0 else v + e[t] * e[t - l] * X[t - l][i] * X[t][j])
    B = inverse(XtX)
    V = [[sum(B[i][a] * S[a][b] * B[b][j] for a in range(k) for b in range(k)) for j in range(k)] for i in range(k)]
    return beta, [math.sqrt(V[i][i]) for i in range(k)], e

def design_row(t, start, ev_len):
    return [1.0, t / 100, 1.0 if start <= t < start + ev_len else 0.0, 1.0 if t >= start + ev_len else 0.0]

def fit(treated, control, days, start, ev_len):
    """log(treated) - log(control) = b0 + b1*t + b2*[event window] + b3*[after window]; b2 as % change."""
    X = [design_row(t, start, ev_len) for t in days]
    y = [math.log(treated[t] + 0.5) - math.log(control[t] + 0.5) for t in days]
    beta, se, _ = ols_nw(X, y)
    b, s = beta[2], se[2]
    return math.exp(b) - 1, math.exp(b - 1.96 * s) - 1, math.exp(b + 1.96 * s) - 1

def cits(treated, control, pre_len=180, ev_len=21, post_end=89, drop=frozenset()):
    days = [t for t in range(-pre_len, post_end + 1) if t not in drop]
    return fit(treated, control, days, 0, ev_len)

def placebo(treated, control, ev_len=21, first=-150, last=-28, step=7):
    """Fake event windows inside the baseline; each fit uses only days before the real event."""
    out = []
    for s in range(first, last - ev_len + 1, step):
        days = range(-180, s + ev_len + 7)
        out.append(fit(treated, control, days, s, ev_len)[0])
    return out

if __name__ == "__main__":
    d = make_region()
    comp = {t: d["W1"][t] + d["W2"][t] for t in DAYS}
    print("site  effect  95% interval        placebo_min  placebo_max  rank")
    for s in ("AP", "BF", "CG"):
        est, lo, hi = cits(d[s], comp)
        pl = placebo(d[s], comp)
        rank = 1 + sum(p >= est for p in pl)
        print(f"{s:4}  {est:+.3f}  ({lo:+.3f}, {hi:+.3f})  {min(pl):+11.3f}  {max(pl):+11.3f}  {rank}/{len(pl) + 1}")
    # secondary outcome: AP median minutes to provider, uncontrolled ITS with linear trend (no comparison)
    rows, y = [], []
    for t in DAYS:
        rows.append([1.0, t / 100, 1.0 if 0 <= t <= 20 else 0.0, 1.0 if t > 20 else 0.0])
        y.append(d["AP_wait"][t])
    beta, se, _ = ols_nw(rows, y)
    print(f"AP wait: event-window change {beta[2]:+.1f} min (95% interval {beta[2] - 1.96 * se[2]:+.1f} to {beta[2] + 1.96 * se[2]:+.1f})")

Expected output (under a second):

site  effect  95% interval        placebo_min  placebo_max  rank
AP    +0.244  (+0.067, +0.449)       -0.190       +0.166  1/16
BF    +0.058  (-0.067, +0.199)       -0.137       +0.255  8/16
CG    +0.008  (-0.093, +0.121)       -0.104       +0.242  9/16
AP wait: event-window change +5.9 min (95% interval +2.6 to +9.2)

Interpretation, written as the dissertation's results chapter would state it:

• RQ1, Alder Point (6 km). Ambulance arrivals during the diversion were an estimated 24.4% higher than expected from the pre-event trend and Wexmoor's concurrent change (95% interval 6.7% to 44.9%). The estimate exceeded all 15 placebo estimates (rank 1 of 16). The generator's true value is 22%; the estimate is close, but the interval is wide — consistent with Lab 3's warning about precision at one ED.
• RQ1, Brookfield (14 km). +5.8% (−6.7% to +19.9%); rank 8 of 16. The data are compatible with no change and with a change of about 20%. The true value was 10%: the study did not detect an effect that exists. The honest statement is "inconclusive", never "no effect".
• RQ1, Calloway (38 km). +0.8% (−9.3% to +12.1%); rank 9 of 16. Compatible with no change; increases above about 12% are unlikely under the model.
• RQ2, distance. The point estimates decline with distance (24.4%, 5.8%, 0.8%), consistent with load transfer concentrated on the nearest ED. Brookfield's and Calloway's intervals overlap heavily, so the data cannot say whether the decline is gradual or confined to the nearest hospital.
• Secondary outcome, Alder Point waiting time. +5.9 minutes during diversion (2.6 to 9.2). This model has no comparison series, because Wexmoor publishes no waiting times; the estimate therefore includes any seasonal crowding. In the synthetic generator, the season adds up to 3 minutes and the true diversion effect is 4 minutes; the estimate combines both. The dissertation must say the waiting-time estimate is confounded by season and likely overstates the diversion's contribution.

6.6 The qualitative strand

6.6.1 Sampling and interviewing

RQ3 asks how load was redistributed, which only people involved can tell. Sampling is purposive: ED leadership at each neighbour, EMS operations and dispatch supervisors, and coalition staff. How many interviews are enough has no single answer. Guest, Bunce and Johnson (2006) found in one study of relatively homogeneous participants that most themes appeared within the first twelve interviews; Malterud, Siersma and Guassora (2016) propose information power: the more relevant information each participant holds for a narrow aim, the fewer participants are needed. The Spillover Study's aim is narrow and its participants hold direct, specific knowledge; ten interviews (I01–I10) across the three roles gave adequate information power, a judgement the dissertation must explain rather than assert.

Interviews followed the revised guide (Chapter 5), anchored to time and information. They were recorded with consent, transcribed, de-identified (names, organization names, unique role titles and dates replaced) and the recordings destroyed after transcription as the consent form stated.

6.6.2 Thematic analysis

Braun and Clarke (2006) describe thematic analysis in six phases: familiarization with the data; generating initial codes; searching for themes; reviewing themes; defining and naming themes; and producing the report. Keep an audit trail: the codebook with definitions and examples, memos explaining each decision, and the versions of the theme structure.

6.6.3 Coder agreement

Whether to measure agreement between coders is debated. O'Connor and Joffe (2020) review the debate and suggest that intercoder reliability can add rigour and transparency when a codebook is applied consistently, while it fits poorly with interpretive approaches in which the researcher's perspective is part of the analysis. The Spillover Study uses a codebook-based approach for the operational questions, so a second coder independently coded a sample of excerpts; agreement was measured with Cohen's kappa (Chapter 1) and every disagreement was discussed and resolved by refining code definitions. The kappa is reported as evidence that the codebook can be applied consistently — not as proof that the themes are true.

6.6.4 Themes (synthetic)

6.7 Integration: the joint display

Fetters, Curry and Creswell (2013) describe integration at the levels of design, methods and interpretation, and the joint display as a way to bring results together. Each row pairs a quantitative result with a qualitative finding and states a meta-inference: confirmation (both strands agree), expansion (one adds to the other) or discordance (they conflict, which requires explanation).

The discordance row is not a weakness of the study; it is a finding. A dissertation that hides discordance to tell a cleaner story loses the most informative part of mixed methods.

6.8 Worked example: the deviation that was nearly hidden (synthetic)

After the main analysis, the candidate noticed that dropping the two regional holidays changed Brookfield's estimate slightly and was tempted to report the version with holidays dropped, which "looked more like the expected gradient". The supervisor's rule: the prespecified model is the primary analysis; any alternative is reported as a robustness check with the reason it was examined, in the deviation log, dated, noting that results had been seen. Chapter 8's specification curve then reports all reasonable choices together, so that no single choice — made after seeing results — carries the conclusion. The episode is typical: the risk of flexible analysis is not dishonesty but the quiet pull of a preferred story (Gelman and Loken, 2014).

6.9 Common errors and limits

• Editing data by hand after freezing, or analysing an extract whose version cannot be identified.
• Fitting an uncontrolled ITS and calling the result an effect when a shared shock is plausible.
• Reporting ordinary least-squares intervals for autocorrelated daily data without checking.
• Converting a placebo rank into "p < 0.05" when the number of placebos makes that impossible.
• Reporting an inconclusive estimate as "no effect".
• Presenting a confounded secondary outcome with the same confidence as the controlled primary outcome.
• Claiming saturation or adequacy of interview numbers without explaining the reasoning.
• Treating a high kappa as proof that themes are valid.
• Hiding discordance between strands.
• Choosing among analysis variants after seeing results and reporting only the preferred one.

The limit of this analysis is that it rests on one event, one comparison region and a model whose assumptions — parallel trends, a linear pre-trend, a step-shaped effect — can be probed but not proven. Chapter 8 tests how much the conclusion depends on them.

Exercises — chapter 6

EX033 — Run and report the main analysis. Objective: use Lab 6 and write results. Inputs: Labs 4 and 6. Constraints: offline; do not change seeds. Steps: (1) run Lab 6 and record the output; (2) write a 200-word results paragraph for RQ1 and RQ2 using estimates, intervals and placebo ranks; (3) write one sentence on the waiting-time result that states its limitation. Expected evidence: output and text. Rubric: 3 = numbers exact, Brookfield described as inconclusive rather than null, gradient stated with the overlap caveat, waiting time flagged as uncontrolled; 2 = one overstatement; 1 = "significant" used as a conclusion without estimates; 0 = none.

EX034 — Lag sensitivity. Objective: test the interval method. Inputs: lab6_its.py. Constraints: change only the Newey–West lag. Steps: (1) compute Alder Point's interval with lags 0, 7 and 14; (2) compute the residual autocorrelation at lags 1 and 7 and the Durbin–Watson statistic; (3) explain why a longer lag does not always widen the interval; (4) state what you would report. Expected evidence: table and paragraph. Rubric: 3 = intervals and diagnostics correct, non-monotonic behaviour explained (Bartlett-weighted sample autocovariances can be negative), prespecified lag reported with sensitivity; 2 = one value wrong; 1 = chooses the narrowest interval; 0 = none.

EX035 — Placebo arithmetic. Objective: interpret rank-based evidence. Inputs: Lab 6's placebo output. Constraints: no new code needed. Steps: (1) state the one-sided rank-based p-value for each site; (2) state how many placebo windows would be needed for the smallest attainable value to reach 0.05 and 0.01, and to fall strictly below them; (3) explain why early placebo windows are noisier than the real fit; (4) propose a change that adds windows without using post-event data. Expected evidence: answers. Rubric: 3 = 1/16, 8/16, 9/16 computed; 19 and 99 windows to reach 0.05 and 0.01, 20 and 100 to fall below; short pre-periods explained; a valid proposal (for example daily rather than weekly start steps, or windows from earlier years if data exist) with its dependence caveat; 2 = one error; 1 = treats placebo p as exact; 0 = none.

EX036 — Repair the secondary outcome. Objective: address a confounded estimate. Inputs: the waiting-time result and §6.5. Constraints: no new data may be invented. Steps: (1) explain the confounding; (2) propose two designs that could remove it with obtainable data; (3) state the claim the current estimate supports; (4) write the sentence for the limitations section. Expected evidence: one page. Rubric: 3 = confounding by season explained, designs feasible (Calloway as a within-region comparison for waiting time given its null arrival effect, or a comparison series from another region's published data), claim limited to "consistent with an increase; magnitude confounded", limitation directional; 2 = one design infeasible; 1 = no limitation; 0 = none.

EX037 — Coder agreement. Objective: apply and interpret kappa for qualitative coding. Inputs: twelve synthetic excerpts coded by two coders with codes DIV (EMS destination decisions), INFO (information and notification), STAFF (staffing and strain), PROC (procedures). Coder A: DIV, INFO, STAFF, DIV, PROC, INFO, INFO, STAFF, DIV, PROC, INFO, STAFF. Coder B: DIV, INFO, STAFF, STAFF, PROC, INFO, DIV, STAFF, DIV, PROC, PROC, STAFF. Excerpt 4: "We had two crews waiting at the door and nobody to take handover." Excerpt 7: "Dispatch didn't know Kestrel was closed, so the crew decided on the road." Excerpt 11: "We didn't know whether to start the paper triage process or wait for word from IT." Constraints: compute by hand or with the Lab 1 function. Steps: (1) compute observed agreement, chance agreement and kappa; (2) for each disagreement, decide whether the codebook definition or the excerpt is the problem; (3) revise one definition. Expected evidence: calculation, analysis and revised definition. Rubric: 3 = κ ≈ 0.67 computed correctly, disagreements diagnosed as boundary problems (excerpts that fit two codes), a revision that allows or rules on double-coding; 2 = calculation correct, diagnosis thin; 1 = raw agreement only; 0 = none.

EX038 — Joint display. Objective: integrate strands. Inputs: §6.5 and §6.6.4. Constraints: at least four rows; at least one discordance. Steps: (1) build the display; (2) state the meta-inference for each row; (3) for each discordance, list explanations and the evidence that could choose among them. Expected evidence: table and paragraph. Rubric: 3 = meta-inferences labelled correctly, discordance explained without letting one strand overrule the other, follow-up evidence specific; 2 = discordance omitted; 1 = strands side by side without inference; 0 = none.

Chapter synthesis

Keep lineage: every extract logged, audited, frozen and used only through scripts. The controlled ITS models the log ratio of a neighbour's arrivals to the comparison region's with a trend, a diversion step and a post-diversion step; exp(β₂) − 1 is the estimate. Check autocorrelation and use robust intervals with a prespecified lag and sensitivity. Placebo ranks are honest evidence whose smallest attainable p-value depends on the number of windows. In the synthetic Spillover Study, Alder Point's arrivals rose by about a quarter during diversion with a wide interval; Brookfield's result is inconclusive even though a real effect existed; Calloway's is compatible with none; the waiting-time estimate is confounded by season. Interviews explain the gradient through road geography and reveal late notification, night-time strain and the link between diversion length and clinical-system recovery. Integrate strands in a joint display and report discordance as a finding.

Continue studying — chapter 06 of 08

← Chapter 05 · Course chapter index · Chapter 07 →

Continue to the next authored lesson.