Chapter 5 — Main study and independent validation
Lesson map
Lesson 05 published. Map, example, EX027.
flowchart TD
N0["Learning outcomes"]
N1["5.1 Main data"]
N2["Lab — ring versus flat"]
N3["5.2 Reading the results"]
N4["5.3 Baseline adjustment"]
N5["5.4 Independent validation"]
N6["5.5 Meaning"]
N7["Secondary outcomes"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX027"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["5.1 Main data"]
N2["Lab — ring versus flat"]
N3["5.2 Reading the results"]
N4["5.3 Baseline adjustment"]
N5["5.4 Independent validation"]
N6["5.5 Meaning"]
N7["Secondary outcomes"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX027"]
[Learning outcomes]
|
v
[5.1 Main data]
|
v
[Lab — ring versus flat]
|
v
[5.2 Reading the results]
|
v
[5.3 Baseline adjustment]
|
v
[5.4 Independent validation]
|
v
[5.5 Meaning]
|
v
[Secondary outcomes]
|
v
[Worked example] --> [Exercise EX027]
Learning outcomes
You will (1) analyse the main study; (2) estimate effects with uncertainty; (3) validate results independently.
5.1 Main data
Six months produce data on 1,000 endpoints at each site. The analysis follows the pre-registered plan. Each site contributes about 1,000 endpoints across six monthly cycles, so each outcome is computed over thousands of update attempts. Endpoints that join or leave a site during the study are counted only for the months they were managed there. The main analysis script is the one written before data collection, run once on the full data. Missing reboot times are handled by the pilot rule, applied identically at both sites and counted in the results.
Lab — ring versus flat
"""Compare ring-based and flat patch deployment on synthetic endpoints with a bootstrap CI (stdlib only)."""
import random, statistics
random.seed(42)
def site(ring, n):
out = []
for _ in range(n):
days = random.gauss(9, 3) if ring else random.gauss(13, 5)
fail = random.random() < (0.012 if ring else 0.031)
out.append((max(1, days), fail))
return out
r, f = site(True, 1000), site(False, 1000)
def p95(x): return sorted(d for d, _ in x)[int(0.95*len(x))]
def fr(x): return sum(fl for _, fl in x) / len(x)
diffs = []
for _ in range(2000):
rs = [random.choice(r) for _ in r]; fs = [random.choice(f) for _ in f]
diffs.append(fr(fs) - fr(rs))
diffs.sort()
print(f"ring: 95th pct {p95(r):.1f} days, failure rate {fr(r):.3f}")
print(f"flat: 95th pct {p95(f):.1f} days, failure rate {fr(f):.3f}")
print(f"failure-rate difference {fr(f)-fr(r):.3f}, 95% CI [{diffs[50]:.3f}, {diffs[1949]:.3f}]")
Run with python3 lab_main.py. Verified output:
ring: 95th pct 14.1 days, failure rate 0.011
flat: 95th pct 20.7 days, failure rate 0.028
failure-rate difference 0.017, 95% CI [0.005, 0.029]
5.2 Reading the results
At the ring site, the 95th percentile is 14.1 days and the failure rate 0.011; at the flat site, 20.7 days and 0.028. The failure-rate difference is 0.017 with a bootstrap 95% confidence interval from 0.005 to 0.029, which excludes zero. Both hypotheses are supported in this synthetic data. The interval is wide relative to the point estimate, so Yusuf reports it in full rather than the difference alone. The phrase about synthetic data matters too: the published figures come from synthetic-structure data with the same distributions, not from the trust's raw records.
5.3 Baseline adjustment
Baseline months showed the flat site was already about two days slower at the 95th percentile. Yusuf reports both the raw difference and the difference after subtracting baseline, which is smaller but still in favour of rings. Reporting both figures lets readers judge how much of the raw difference the rings explain. The baseline-adjusted difference is the one Yusuf leads with in the abstract, because it accounts for a gap that existed before the intervention. The raw figure stays in the results table for completeness.
5.4 Independent validation
A second analyst at Calder, given only the pre-registered plan and the data, reproduces the figures independently. The second analyst wrote her own code from the plan, so a bug in Yusuf's script would show up as a mismatch. Where her figures and his agree, the thesis says so; the comparison table is in an appendix with both scripts. This is the reproducibility check examiners will ask about first. She received no access to Yusuf's earlier outputs until her own run had finished.
5.5 Meaning
Fewer failures matter because a failed update on a clinical endpoint can stop work on a ward; faster coverage reduces the window in which known vulnerabilities are exposed. For Harrowgate's managers, the practical meaning is in the examiner's conversion: even at the lower bound of the interval, rings would prevent dozens of failed updates a year across 1,000 endpoints. The faster 95th percentile matters for clinical endpoints that are hard to take offline, since they are often the last to be patched.
Secondary outcomes
Pre-registered secondary outcomes include help-desk tickets per update and the share of endpoints never patched within 30 days. Both favour rings in the synthetic data, but Yusuf reports them as secondary and does not let them change the main conclusion.
Reporting standards
Figures report counts as well as rates, and every number in the text links to the analysis script line that produced it.
Sensitivity analysis
Yusuf repeats the analysis excluding the 3% of endpoints with imputed reboot times; the results barely change, which supports the main finding.
Worked example
An examiner asks whether the 0.005 lower bound is practically meaningful. Yusuf converts it: at 1,000 endpoints and 12 monthly cycles, even the lower bound means about 60 fewer failed updates a year.
Synthetic case. Synthetic.
Errors and limits
Site-level designs cannot rule out unmeasured site differences.
Exercises
- EX027 — Lab results. Report the main figures for both sites. Rubric: 3 = ring 14.1/0.011; flat 20.7/0.028; 2 = two; 1 = one; 0 = none.
- EX028 — Confidence interval. Give the failure-rate difference and CI. Rubric: 3 = 0.017 [0.005, 0.029]; 2 = partial; 1 = vague; 0 = none.
- EX029 — Baseline adjustment. How was the baseline difference handled? Rubric: 3 = report raw and adjusted; 2 = partial; 1 = vague; 0 = none.
- EX030 — Validation. How was validation done? Rubric: 3 = second analyst from plan and data; 2 = partial; 1 = vague; 0 = none.
- EX031 — Practical meaning. How did Yusuf express the lower bound practically? Rubric: 3 = ~60 fewer failures a year; 2 = partial; 1 = vague; 0 = none.
- EX032 — Remaining threat. Which threat remains? Rubric: 3 = unmeasured site differences; 2 = partial; 1 = vague; 0 = none.
Synthesis
Report effects with uncertainty, baseline adjustment and independent validation.
Continue studying — chapter 05 of 08
← Chapter 04 (draft) · Course chapter index · Chapter 06 (draft) →