Chapter 6 — Reproducible collection and outcome analysis

Lesson map

Lesson 06 published. Map, example, EX033.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["6.1 Collecting data with lineage"]
  N2["6.2 The main-study data synthetic"]
  N3["6.3 The primary analysis"]
  N4["6.4 Practical activity: the main analysis"]
  N5["6.5 What robustness shows and does not show"]
  N6["6.6 Missing data and the withdrawn analyst"]
  N7["6.7 The qualitative strand"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX033"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["6.1 Collecting data with lineage"]
  N2["6.2 The main-study data synthetic"]
  N3["6.3 The primary analysis"]
  N4["6.4 Practical activity: the main analysis"]
  N5["6.5 What robustness shows and does not show"]
  N6["6.6 Missing data and the withdrawn analyst"]
  N7["6.7 The qualitative strand"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX033"]

[Learning outcomes and connection]
    |
    v
[6.1 Collecting data with lineage]
    |
    v
[6.2 The main-study data synthetic]
    |
    v
[6.3 The primary analysis]
    |
    v
[6.4 Practical activity: the main analysis]
    |
    v
[6.5 What robustness shows and does not show]
    |
    v
[6.6 Missing data and the withdrawn analyst]
    |
    v
[6.7 The qualitative strand]
    |
    v
[Worked example] --> [Exercise EX033]

Learning outcomes and connection

After this chapter you should be able to:

1. Write a data dictionary and cleaning log that let another person trace every derived value to its raw source.
2. Carry out the predeclared paired analysis of a crossover study with a bootstrap interval and a permutation test, and check period effects.
3. Report secondary outcomes — time, over-triage and agreement with a reference standard — so that costs are visible beside benefits.
4. Test alternative outcome definitions and say what robustness does and does not show.
5. Analyse debrief interviews thematically, check coding consistency and integrate the two strands in a joint display.

The chapter builds on the revised instruments of Chapter 5 and prepares Chapter 7, where results become claims.

6.1 Collecting data with lineage

Every value in the analysis should have a lineage: where it came from, what was done to it and by which code. Three documents make that possible.

The data dictionary defines each variable:

Lesson table; scroll horizontally to read all columns →

The cleaning log records every transformation in order, with the script that performed it and the number of rows affected: "Step 3: removed 12 rows of A17 period 2 placeholders (withdrawal); script clean.py line 41; rows 576 → 564." The raw data are kept read-only. A reader with the raw files, the code and the log should arrive at exactly the analysis dataset.

6.2 The main-study data (synthetic)

Twenty-four analysts enrolled and were allocated in order to the four sequences. Analyst A17 withdrew after period 1 for personal reasons, giving 23 analysts with both periods. Each completed period has twelve decisions, of which eight are on high-impact cases. The generator below produces the synthetic dataset; in a real study this file would be replaced by the cleaned export, and the analysis script would be unchanged. The generator encodes these assumptions, which you should treat as the "truth" of the synthetic world: analysts differ in a stable bias; quiet-impact cases are under-rated by about one severity level under usual judgement and by about 0.4 levels with the rubric; noisy cases are unaffected by the rubric; there is a small practice effect in period 2; and the rubric increases decision time by about 18%. Save it as lab6_data.py.

# lab6_data.py - generator for the synthetic main study of the Triage Rubric Study (offline, stdlib only)
# Save under this name: Lab 6 and Lab 8 import make_study() from it. Every value is fictional.
# 24 volunteer analysts (A01-A24) triage 24 fictional cases: set A (C01-C12) and set B (C13-C24), each with
# eight high-impact cases (reference severity 3-4), half of them "quiet" (high business impact, low alert noise).
# Four sequences balance case set and condition order: U = usual judgement, R = SEV-4 rubric.
import random
SEQUENCES = {1: (("U", "A"), ("R", "B")), 2: (("U", "B"), ("R", "A")),
             3: (("R", "A"), ("U", "B")), 4: (("R", "B"), ("U", "A"))}
def make_cases(seed=11):
    rng = random.Random(seed); cases = {}
    for s, offset in (("A", 0), ("B", 12)):
        refs = [4, 4, 3, 3, 4, 3, 3, 3, 2, 2, 1, 1]
        quiet = [1, 0, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0]
        for i, (ref, q) in enumerate(zip(refs, quiet)):
            cases[f"C{offset + i + 1:02d}"] = dict(set=s, ref=ref, quiet=q, base_s=rng.uniform(150, 330))
    return cases
def make_study(seed=2031, withdrawn=("A17",), quiet_usual=-1.0, quiet_rubric=-0.4, period2=0.1, noise=0.55):
    rng = random.Random(seed); cases = make_cases(); rows = []
    for a in range(1, 25):
        aid = f"A{a:02d}"; seq = (a - 1) % 4 + 1
        bias = rng.gauss(0, 0.35); speed = rng.lognormvariate(0, 0.25)
        for period, (cond, cset) in enumerate(SEQUENCES[seq], start=1):
            if aid in withdrawn and period == 2:
                continue
            for cid, c in cases.items():
                if c["set"] != cset:
                    continue
                latent = c["ref"] + bias + c["quiet"] * (quiet_rubric if cond == "R" else quiet_usual) \
                         + (period2 if period == 2 else 0) + rng.gauss(0, noise)
                rating = min(4, max(1, round(latent)))
                secs = c["base_s"] * speed * (1.18 if cond == "R" else 1.0) * rng.lognormvariate(0, 0.2)
                rows.append(dict(analyst=aid, seq=seq, period=period, cond=cond, case=cid, ref=c["ref"],
                                 quiet=c["quiet"], rating=rating, seconds=round(secs)))
    return rows

The generator's seed (2031) was chosen when the book was written so that the synthetic dataset behaves like a typical draw: for some other seeds, chance produces a sizeable apparent "effect" in noisy cases where the generator has none — itself a lesson about subgroup analyses in small studies.

6.3 The primary analysis

The estimand (Chapter 1) is the mean, across analysts, of each analyst's under-triage proportion with the rubric minus their proportion without it. The analysis follows the pre-analysis plan:

• Per-analyst differences. Computing one difference per analyst respects the clustering of decisions within analysts: the 23 differences, not the 368 high-impact decisions, are the independent units.
• Bootstrap interval. Resample the 23 differences with replacement many times and take the 2.5th and 97.5th percentiles of the resampled means (Efron 1979). The percentile bootstrap is simple and transparent; with 23 units it is approximate, and the dissertation says so.
• Sign-flip permutation test. If the rubric had no effect, each analyst's difference would be equally likely to be positive or negative. Randomly flipping signs many times gives the distribution of the mean under that null; the p-value is the share of flipped means at least as extreme as the observed one.
• Period check. Compare within-analyst changes from period 1 to period 2 between analysts who went usual→rubric and those who went rubric→usual. Half their sum estimates the period effect.

Secondary outcomes are computed on the same 23 analysts: quadratic-weighted kappa between rating and reference (Cohen 1968), median decision seconds, and over-triage of low-impact cases (rating above reference, among references 1–2).

6.4 Practical activity: the main analysis

Save the analysis as lab6_analysis.py in the same folder as lab6_data.py and run it.

# Lab 6 - main analysis of the synthetic Triage Rubric Study (offline, stdlib only; imports lab6_data.py)
import random, statistics as st
from lab6_data import make_study
ROWS = make_study()
def under(r, levels=1): return r["rating"] <= r["ref"] - levels
def per_analyst(rows, cond, keep=lambda r: r["ref"] >= 3, levels=1):
    out = {}
    for r in rows:
        if r["cond"] == cond and keep(r):
            out.setdefault(r["analyst"], []).append(under(r, levels))
    return {a: sum(v) / len(v) for a, v in out.items()}
def paired(rows, **kw):
    R, U = per_analyst(rows, "R", **kw), per_analyst(rows, "U", **kw)
    both = sorted(set(R) & set(U))
    return [R[a] - U[a] for a in both], R, U, both
def boot_ci(d, reps=4000, seed=1):
    rng = random.Random(seed); m = sorted(st.mean(rng.choices(d, k=len(d))) for _ in range(reps))
    return m[int(0.025 * reps)], m[int(0.975 * reps) - 1]
def signflip_p(d, reps=20000, seed=2):
    rng = random.Random(seed); obs = abs(st.mean(d))
    hits = sum(abs(st.mean([x if rng.random() < 0.5 else -x for x in d])) >= obs - 1e-12 for _ in range(reps))
    return (hits + 1) / (reps + 1)
def wkappa(pairs, k=4):
    n = len(pairs); w = lambda i, j: 1 - ((i - j) / (k - 1)) ** 2
    po = sum(w(a, b) for a, b in pairs) / n
    pa = [sum(a == c for a, _ in pairs) / n for c in range(1, k + 1)]
    pb = [sum(b == c for _, b in pairs) / n for c in range(1, k + 1)]
    pe = sum(pa[i] * pb[j] * w(i + 1, j + 1) for i in range(k) for j in range(k))
    return (po - pe) / (1 - pe)
if __name__ == "__main__":
    d, R, U, both = paired(ROWS)
    print(f"analysts enrolled 24, with both periods {len(both)} (A17 withdrew after period 1)")
    print(f"primary: under-triage of high-impact cases  usual {st.mean(U[a] for a in both):.3f}  rubric {st.mean(R[a] for a in both):.3f}")
    lo, hi = boot_ci(d)
    print(f"  paired difference (rubric - usual) {st.mean(d):+.3f}  95% bootstrap CI {lo:+.3f} to {hi:+.3f}  sign-flip p = {signflip_p(d):.4f}")
    print(f"  analysts better / same / worse with rubric: {sum(x < 0 for x in d)} / {sum(x == 0 for x in d)} / {sum(x > 0 for x in d)}")
    # period (learning) effect: within-analyst period-2 minus period-1 rate, by condition order
    def rate(a, p): v = [under(r) for r in ROWS if r["analyst"] == a and r["period"] == p and r["ref"] >= 3]; return sum(v) / len(v)
    pd_ur = [rate(a, 2) - rate(a, 1) for a in both if int(a[1:]) % 4 in (1, 2)]
    pd_ru = [rate(a, 2) - rate(a, 1) for a in both if int(a[1:]) % 4 in (3, 0)]
    print(f"period check: P2-P1 when U->R {st.mean(pd_ur):+.3f}; when R->U {st.mean(pd_ru):+.3f}; period effect {(st.mean(pd_ur) + st.mean(pd_ru)) / 2:+.3f}")
    for q, label in ((1, "quiet"), (0, "noisy")):
        dq, *_ = paired(ROWS, keep=lambda r, q=q: r["ref"] >= 3 and r["quiet"] == q)
        print(f"  subgroup {label:<5} high-impact cases: difference {st.mean(dq):+.3f}")
    for cond in ("U", "R"):
        k = wkappa([(r["ref"], r["rating"]) for r in ROWS if r["cond"] == cond and r["analyst"] in both])
        s = st.median(r["seconds"] for r in ROWS if r["cond"] == cond and r["analyst"] in both)
        o = sum(r["rating"] > r["ref"] for r in ROWS if r["cond"] == cond and r["analyst"] in both and r["ref"] <= 2) / \
            sum(1 for r in ROWS if r["cond"] == cond and r["analyst"] in both and r["ref"] <= 2)
        print(f"secondary {cond}: weighted kappa vs reference {k:.3f}  median seconds {s:.0f}  over-triage of low-impact {o:.3f}")
    print("robustness of the primary contrast:")
    for name, kw in (("under by 2+ levels", dict(levels=2)), ("include reference-2 cases", dict(keep=lambda r: r["ref"] >= 2)),
                     ("critical cases only (ref 4)", dict(keep=lambda r: r["ref"] == 4))):
        dd, *_ = paired(ROWS, **kw)
        lo2, hi2 = boot_ci(dd)
        print(f"  {name:<28} {st.mean(dd):+.3f}  CI {lo2:+.3f} to {hi2:+.3f}")

Expected output:

analysts enrolled 24, with both periods 23 (A17 withdrew after period 1)
primary: under-triage of high-impact cases  usual 0.538  rubric 0.364
  paired difference (rubric - usual) -0.174  95% bootstrap CI -0.245 to -0.098  sign-flip p = 0.0005
  analysts better / same / worse with rubric: 19 / 1 / 3
period check: P2-P1 when U->R -0.148; when R->U +0.198; period effect +0.025
  subgroup quiet high-impact cases: difference -0.370
  subgroup noisy high-impact cases: difference +0.022
secondary U: weighted kappa vs reference 0.623  median seconds 236  over-triage of low-impact 0.261
secondary R: weighted kappa vs reference 0.755  median seconds 281  over-triage of low-impact 0.239
robustness of the primary contrast:
  under by 2+ levels           -0.082  CI -0.120 to -0.049
  include reference-2 cases    -0.170  CI -0.239 to -0.104
  critical cases only (ref 4)  -0.246  CI -0.377 to -0.101

How to read it.

• Primary result. Averaged across the 23 analysts, the under-triage proportion for high-impact cases was 0.538 with usual judgement and 0.364 with the rubric. The mean within-analyst difference is −0.174 (−17.4 percentage points), with a 95% bootstrap interval from −24.5 to −9.8 points and a sign-flip p-value of 0.0005 (the smallest value 20,000 flips can give is about 0.00005, so this is not at a floor). Nineteen analysts did better with the rubric, one the same and three worse. These proportions are high because the case sets were deliberately enriched with quiet-impact cases; they are not estimates of how often under-triage occurs in practice.
• Period check. Analysts who went usual→rubric improved by 14.8 points from period 1 to 2; those who went rubric→usual worsened by 19.8 points. The estimated period effect is +2.5 points (slightly more under-triage in period 2) — small, and in the opposite direction to the generator's practice effect, which is a reminder that a check with 23 analysts is itself noisy. Because order was balanced, a period effect of this size does not bias the primary contrast. The asymmetry between the two orders (−14.8 versus +19.8) is also what carryover would look like if rubric-first analysts had kept some benefit; here it is within noise, and Chapter 8's sequence-level check returns to it.
• Mechanism subgroup. The reduction is concentrated in quiet-impact cases (−37.0 points); on noisy cases the difference is +2.2 points, consistent with no effect. This was predeclared (Chapter 4) and matches the prediction of the situation-awareness account.
• Costs and agreement. Weighted kappa against the reference rose from 0.623 to 0.755. Median decision time rose from 236 to 281 seconds, about 45 seconds per case. Over-triage of low-impact cases did not rise (0.261 usual, 0.239 rubric), which argues against the rival "the rubric simply pushes every rating up".
• Robustness. Counting only errors of two or more levels, the difference is −8.2 points (interval −12.0 to −4.9); including severity-2 cases, −17.0; critical cases only, −24.6 with a wider interval. All point the same way.

6.5 What robustness shows and does not show

Robustness checks show that a conclusion does not depend on one arbitrary choice. They do not show that the conclusion is correct: if all specifications share a flaw — a biased reference standard, say, or cases unlike real ones — they will agree with each other and all be wrong. Report robustness as "the direction and approximate size held across the predeclared alternatives", not as "the result is robust", and keep the shared assumptions in view. Chapter 8 widens the set of alternatives into a specification curve.

Two other temptations deserve naming. The first is to compute a pooled proportion over all decisions and report it with a binomial interval: that treats 368 decisions as independent and produces an interval that is too narrow. The second is to drop the three analysts who did worse with the rubric because they "did not engage with it". Unless fidelity checks recorded non-use (a deviation, with a predeclared rule), they stay in.

6.6 Missing data and the withdrawn analyst

A17 completed period 1 (usual judgement, sequence 1) and withdrew. The primary analysis uses analysts with both periods, as predeclared. Is that biased? It is unbiased if A17's withdrawal is unrelated to what their rubric result would have been — plausible here, since they withdrew for personal reasons before seeing the rubric — but the assumption should be stated. A sensitivity analysis can bound the effect: give A17 the worst plausible difference (no improvement, or the largest worsening observed) and the best, and report the range of the mean. With one withdrawal out of 24 the range is narrow; with six it might not be. The general framework for thinking about missing outcome data in trials, including sensitivity analyses under different assumptions, is set out by White et al. (2011).

6.7 The qualitative strand

6.7.1 Sampling and interviewing

RQ3 asks how analysts describe their reasoning when the rubric changed, or failed to change, their rating. After all sessions were complete, twelve analysts were invited for 30-minute debrief interviews, chosen purposively: four whose under-triage fell most, four whose results barely changed, and the three who did worse plus one more from the middle. The interviewer showed each analyst two of their own decisions — one where their usual and rubric ratings of matched cases differed, one where they did not — and asked them to reconstruct their reasoning. Twelve interviews is a judgement, not a rule; Malterud, Siersma and Guassora (2016) argue for sizing by "information power", which depends on the narrowness of the aim, the specificity of the sample and the quality of dialogue.

6.7.2 Thematic analysis

Transcripts were analysed with reflexive thematic analysis following Braun and Clarke (2006): familiarization, initial coding, searching for themes, reviewing, defining and naming, and writing. Codes were generated from the data rather than from the hypothesis, though the analyst knew the hypothesis and recorded that in a reflexive memo. Every code and theme change was dated in an audit trail.

6.7.3 Coding consistency

In reflexive thematic analysis, inter-coder agreement statistics are contested, because the method treats coding as interpretive. For a master's dissertation, a practical compromise is for a second person to code three transcripts independently with the codebook, then discuss disagreements to sharpen code definitions, reporting the process and its effect on the codebook rather than treating a kappa as a validity certificate.

6.7.4 Themes (synthetic)

1. "The quiet ones look like housekeeping." Analysts described quiet-impact cases as resembling routine maintenance noise; the rubric's question about which service was affected "made me look up what the job actually did" (A05).
2. "The rubric slowed me down where I already knew." Several experienced analysts said the rubric added time on obvious cases without changing their answer — consistent with the time cost and the null noisy-case result.
3. "I trusted my first read." Two of the three analysts who did worse said they completed the rubric after already deciding and adjusted answers to match their first impression — a fidelity concern that the observer form did not capture.
4. "It gave me words for escalating." Analysts described the rubric as justification they could cite to a shift lead when escalating a case with few alerts.

6.8 Integration: the joint display

Mixed-methods integration means more than placing two results side by side. A joint display (O'Cathain, Murphy and Nicholl 2010 describe this and other techniques) lines up quantitative and qualitative findings on the same question and states whether they converge, complement or are discordant.

Lesson table; scroll horizontally to read all columns →

Discordance, when it appears, is reported, not resolved by letting one strand win.

6.9 Worked example: the deviation that was nearly hidden (synthetic)

While checking fidelity notes, the student found that observer 1 had written "sheet completed after rating?" beside two sessions — the same two analysts who, in interview, described completing the rubric after deciding. The observer form had no field for this; the notes were in the margin. The temptation was to exclude those analysts as non-adherent, which would have improved the result. Instead, the student logged the observation as a deviation discovered after data collection, kept the analysts in the primary analysis as predeclared, and reported an exploratory analysis excluding them, labelled as such, alongside a recommendation that future studies time-stamp rubric completion. The dissertation is stronger for it: the finding about how rubrics fail is itself a contribution.

6.10 Common errors and limits

• Treating decisions as independent and reporting intervals that are too narrow.
• Omitting the period and carryover checks in a crossover.
• Reporting the benefit without the costs (time, over-triage).
• Calling a result "robust" because predeclared variations agree, without naming the assumptions they share.
• Dropping participants whose results are inconvenient without a predeclared fidelity rule.
• Presenting a subgroup found after looking at the data as if it were predeclared.
• Quoting interview excerpts selectively to support the quantitative result; ignoring discordant themes.
• Copying numbers into the text by hand instead of from output files.

The limit of the analysis is the design: precise estimates on fictional cases are still estimates about fictional cases. Chapter 7 states that limit with its direction.

Exercises — chapter 6

EX033 — Data dictionary and cleaning log. Objective: make lineage explicit. Inputs: §6.1. Constraints: add two variables the study should have recorded but did not. Steps: (1) add the variables to the dictionary (for example, rubric completion time stamp; observer 2 time); (2) write the first five steps of a cleaning log for the raw answer sheets. Expected evidence: the table and log. Rubric: 3 = variables well defined with sources, log steps specific with row counts and scripts; 2 = one vague entry; 1 = generic; 0 = none.

EX034 — Run Lab 6 and interpret. Objective: compute and explain the primary analysis. Inputs: Lab 6. Constraints: offline. Steps: (1) run it and record the output; (2) write two sentences reporting the primary result with its interval; (3) explain in one sentence why the pooled 368-decision proportion would give a misleadingly narrow interval. Expected evidence: outputs and sentences. Rubric: 3 = numbers exact, interval and p-value correctly worded, clustering explained; 2 = one wording error ("proves"); 1 = not run; 0 = none.

EX035 — Withdrawal sensitivity. Objective: bound the effect of the missing analyst. Inputs: Lab 6 output. Constraints: assume A17's difference could be anything from the largest worsening observed among the 23 analysts to the largest improvement. Steps: (1) compute the observed minimum and maximum per-analyst differences (modify the lab to print them); (2) compute the mean of 24 differences under each extreme; (3) state the range. Expected evidence: the calculation. Rubric: 3 = extremes and resulting means correct, interpretation that one withdrawal cannot change the conclusion, assumption stated; 2 = arithmetic slip; 1 = no numbers; 0 = none.

EX036 — Period and carryover. Objective: interpret crossover checks. Inputs: the period check line. Constraints: use only Lab 6 output. Steps: (1) explain how −0.148 and +0.198 combine into the period effect; (2) explain what pattern carryover would produce; (3) say whether these numbers are evidence of carryover. Expected evidence: a paragraph. Rubric: 3 = period effect computed as half the sum, carryover pattern described (rubric-first analysts' usual period better than usual-first analysts' usual period), conclusion cautious given 23 analysts; 2 = one confusion; 1 = restatement; 0 = none.

EX037 — Codebook entry. Objective: define a qualitative code precisely. Inputs: Theme 3. Constraints: use the fields name, definition, include when, exclude when, example. Steps: write entries for two codes that could underlie Theme 3, and one for a code that a second coder might confuse with them. Expected evidence: three entries. Rubric: 3 = definitions distinguish "post-hoc completion" from "disagreeing with the rubric" from "time pressure", include/exclude rules usable; 2 = overlapping; 1 = labels only; 0 = none.

EX038 — Joint display with discordance. Objective: integrate honestly. Inputs: the joint display in §6.8. Constraints: add one row where the strands could be discordant. Steps: (1) propose a plausible discordant finding (for example, interviews say the rubric helped most on critical cases, while the critical-only estimate is imprecise); (2) write the integration cell; (3) say what further data would resolve it. Expected evidence: the row and a paragraph. Rubric: 3 = discordance stated without resolving it by fiat, further data specific; 2 = discordance explained away; 1 = no discordance; 0 = none.

Chapter synthesis

Collect data with lineage: a data dictionary, a cleaning log and read-only raw files. Analyse a crossover within participants, with a resampling interval, a permutation test and period and carryover checks. Report costs beside benefits and test predeclared alternative outcomes, remembering that agreement among specifications is not proof of correctness. Handle missing data by stating assumptions and bounding their effect. Analyse interviews with a documented, reflexive method and integrate the strands in a joint display that reports convergence, complementarity and discordance alike.