Chapter 5 — Main study and independent validation
Lesson map
Lesson 05 published. Map, example, EX027.
flowchart TD
N0["Learning outcomes"]
N1["5.1 Main study"]
N2["Lab — bootstrap interval"]
N3["5.2 Lab result"]
N4["5.3 Main results"]
N5["5.4 Independent validation"]
N6["5.5 Interviews"]
N7["5.6 Integration"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX027"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["5.1 Main study"]
N2["Lab — bootstrap interval"]
N3["5.2 Lab result"]
N4["5.3 Main results"]
N5["5.4 Independent validation"]
N6["5.5 Interviews"]
N7["5.6 Integration"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX027"]
[Learning outcomes]
|
v
[5.1 Main study]
|
v
[Lab — bootstrap interval]
|
v
[5.2 Lab result]
|
v
[5.3 Main results]
|
v
[5.4 Independent validation]
|
v
[5.5 Interviews]
|
v
[5.6 Integration]
|
v
[Worked example] --> [Exercise EX027]
Learning outcomes
You will (1) run the main quantitative study; (2) estimate uncertainty with the bootstrap; (3) validate results with an independent dataset and analyst; (4) integrate qualitative findings.
5.1 Main study
The main study covers 340 matched project pairs across three vulnerable packages. The analysis plan, pre-registered, uses the difference in median exposure with a bootstrap interval. Three vulnerable packages rather than one guard against a result that depends on quillog's particular fix. The analysis plan was registered before the main collection, and the matching variables and exclusion rule for abandoned projects are applied exactly as in the protocol. Lockfiles and bot pull-request dates are collected as the pilot required. Abandoned projects get their own table, outside the matched comparison.
Lab — bootstrap interval
"""Bootstrap 95% interval for the difference in median exposure days (no bot minus bot), seed 7."""
import random
from statistics import median
bot = [3, 5, 2, 40, 6, 4]; nobot = [31, 12, 88, 45, 20, 160]
rng = random.Random(7); diffs = []
for _ in range(5000):
diffs.append(median(rng.choices(nobot, k=6)) - median(rng.choices(bot, k=6)))
diffs.sort()
print("observed difference:", median(nobot) - median(bot))
print("95% interval:", diffs[124], "to", diffs[4874])
Run with python3 lab_bootstrap.py. Verified output:
observed difference: 33.5
95% interval: 8.0 to 118.5
5.2 Lab result
On the pilot data, the observed difference is 33.5 days (38 minus 4.5) and the 95% bootstrap interval runs from 8.0 to 118.5 days (5,000 resamples, seed 7). The interval excludes zero but is very wide; small samples give imprecise estimates, which is why the main study is larger. The point estimate comes from only twelve projects, so a few long-exposure projects can shift it greatly. Yusuf includes the pilot interval to show readers how much the main study's sample size narrows it. The seed and resample count are recorded so the interval can be regenerated exactly.
5.3 Main results
In the synthetic main study, the median difference is 21 days with a 95% interval of 15 to 28 days. H1 is supported. The interval is much narrower than the pilot's, which reflects 340 matched pairs instead of twelve projects. Yusuf reports the median difference first and leaves causal language for later, because matched observational data cannot rule out that careful projects both adopt bots and update faster. The within-project comparison added after critique addresses that concern. Prof. Okafor's EPSS-weighted sensitivity analysis left this conclusion unchanged.
5.4 Independent validation
A second analyst, given only the protocol, recomputes exposure for a random 10% of pairs; 33 of 34 match exactly, the other differs by one day due to a time-zone boundary. The protocol is clarified to use UTC dates. The single mismatch came from a date near midnight being read in different time zones by the two analysts. Clarifying the protocol to use UTC dates removes the ambiguity for any later replication. The second analyst worked from the protocol alone, so agreement also shows that the written definitions are sufficient.
5.5 Interviews
Twenty maintainers explain lag: test failures on bot updates (9), fear of breaking changes without tests (6), pinned transitive dependencies (3), no time (2). Coded with kappa 0.78. The counts add to twenty, one main reason coded per maintainer. Test failures and fear of breaking changes together account for most answers, which matches H2. The panel's check against reverted-upgrade history was applied to each interviewee, so the stated reasons are compared with what their projects actually did. At 0.78, kappa clears the protocol's target of 0.70 for interview coding as well.
5.6 Integration
Projects with long exposure despite bots mostly had weak tests, matching the interview themes. The joint display links these halves: exposure distributions on one side and interview themes on the other, with the test-failure finding connecting them. Yusuf states the link as consistent rather than proven, since interviewees and long-exposure projects need not be the same projects. The EPSS-weighted sensitivity analysis gave the same picture. Yusuf avoids reading the link as proof that tests cause lag, and the thesis presents it as a pattern for future work to test directly.
Reporting standards
Results are reported with sample sizes, effect sizes, intervals and every analysis deviation from the pre-registration. Yusuf's one deviation, adding UTC dates, is listed with its reason. Readers can then judge the evidence without guessing what was tried and dropped.
Worked example
Yusuf presents a joint display: exposure distributions on the left, interview themes with counts on the right, linked by the test-failure finding. Prof. Okafor suggests weighting exposure by EPSS scores as a sensitivity analysis; results hold.
Synthetic case. Main-study numbers are synthetic.
Errors and limits
Bootstrap intervals assume the sample represents the population; matched pairs from one ecosystem limit this.
Exercises
- EX027 — Lab numbers. Report the lab's observed difference and interval. Rubric: 3 = 33.5; 8.0 to 118.5; 2 = partial; 1 = vague; 0 = none.
- EX028 — Wide interval. Why is the pilot interval wide? Rubric: 3 = small samples; 2 = partial; 1 = vague; 0 = none.
- EX029 — Main result. What was the main-study result? Rubric: 3 = 21 days, 15–28; H1 supported; 2 = partial; 1 = vague; 0 = none.
- EX030 — Validation outcome. What did independent validation find? Rubric: 3 = 33/34 match; one day off from time zone; UTC rule; 2 = partial; 1 = vague; 0 = none.
- EX031 — Top lag reason. What was the most common lag reason? Rubric: 3 = test failures on bot updates (9); 2 = partial; 1 = vague; 0 = none.
- EX032 — Sensitivity analysis. What sensitivity analysis was run? Rubric: 3 = EPSS-weighted exposure; held; 2 = partial; 1 = vague; 0 = none.
Synthesis
Estimate with uncertainty, validate independently and integrate the human explanations.
Continue studying — chapter 05 of 08
← Chapter 04 (draft) · Course chapter index · Chapter 06 (draft) →