Chapter 5 — Main study and independent validation
Lesson map
Lesson 05 published. Map, example, EX027.
flowchart TD
N0["Learning outcomes and connection"]
N1["5.1 Paired design"]
N2["5.2 Three uncertainties"]
N3["5.3 Independent validation with frozen rules"]
N4["5.4 Blinded adjudication"]
N5["5.5 Practical activity — Lab 5: paired bootstrap"]
N6["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX027"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["5.1 Paired design"]
N2["5.2 Three uncertainties"]
N3["5.3 Independent validation with frozen rules"]
N4["5.4 Blinded adjudication"]
N5["5.5 Practical activity — Lab 5: paired bootstrap"]
N6["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX027"]
[Learning outcomes and connection]
|
v
[5.1 Paired design]
|
v
[5.2 Three uncertainties]
|
v
[5.3 Independent validation with frozen rules]
|
v
[5.4 Blinded adjudication]
|
v
[5.5 Practical activity — Lab 5: paired bootstrap]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX027]
Learning outcomes and connection
After this chapter you can (1) run a paired comparison of two inventory methods on the same items; (2) separate sampling uncertainty, measurement uncertainty and model uncertainty; (3) validate on an independent corpus with frozen rules; (4) run blinded adjudication and report agreement; and (5) write an honest results paragraph. It executes the design that Chapter 4 piloted and supplies the results Chapter 6 will attack.
5.1 Paired design
Both methods are applied to the same adjudicated items, so the quantity of interest is a paired difference: for each quantum-vulnerable use, did static analysis find it, and did dual-source analysis find it? Pairing removes between-item variability — some uses are simply hard to find — and gives a much tighter interval than comparing two independent recall estimates. Because dual-source is a union that includes the static findings, the per-item difference can only be 0 or +1 for recall; precision, by contrast, can fall, which is why H2 exists.
5.2 Three uncertainties
Sampling uncertainty is what a bootstrap interval captures: had a different set of uses been adjudicated, the estimate would differ. Measurement uncertainty comes from adjudication: two engineers may disagree on whether a use exists, so report agreement (κ) and run a sensitivity analysis in which disputed items are counted both ways. Model uncertainty comes from analytic choices — the canonicalization rule, the observation window, the codebook definition of "use". Report a small specification curve: the main estimate under each defensible alternative. A conclusion that holds across all three is robust; one that holds only under one specification is a finding about that specification.
5.3 Independent validation with frozen rules
The Marrow Logistics corpus was never examined during development. The fusion rules, thresholds and codebook are frozen and hashed (SHA-256 of the configuration file recorded in the thesis log) before Marrow data are opened. Marrow differs deliberately: fewer vendor appliances, more embedded handheld scanners in warehouses, and an SSH-heavy operations team. The mechanism of Chapter 3 therefore predicts a smaller aggregate gain — a successful prediction of reduced benefit is evidence for the mechanism, not against the method.
Record the date Marrow data were first opened next to the frozen configuration hash.
5.4 Blinded adjudication
Engineers E01–E10 (synthetic) receive each candidate use with tool provenance removed and randomized order, plus owner documentation. Two engineers adjudicate every item; a third resolves disagreements. On the main study, synthetic agreement is κ = 0.78 after the codebook amendment from the pilot. Items no engineer can determine are coded unknown: 9% of HRHN assets and 14% of Marrow assets, reported alongside recall rather than dropped.
Report agreement per asset class as well as overall, because adjudicators may find appliances harder to judge than in-house code. Keep the third engineer's resolutions in a separate file, so that a reader can see how many items needed one and whether any class was disputed more than the others.
5.5 Practical activity — Lab 5: paired bootstrap
# Lab 5 (synthetic): paired bootstrap CI for the recall gain (dual - static) on two corpora.
import random
def corpus(seed, mix):
rng = random.Random(seed); items = []
probs = {"code": (0.90, 0.55), "appliance": (0.20, 0.80), "embedded": (0.30, 0.10), "rest": (0.70, 0.0)}
for cls, n in mix.items():
ps, pp = probs[cls]
for _ in range(n):
s = rng.random() < ps; p = rng.random() < pp
items.append((s, s or p))
return items
def gain_ci(items, reps=2000, seed=1):
rng = random.Random(seed); n = len(items)
point = sum(d - s for s, d in items) / n
boots = sorted(sum(d - s for s, d in (items[rng.randrange(n)] for _ in range(n))) / n for _ in range(reps))
return point, boots[int(0.025 * reps)], boots[int(0.975 * reps)]
for name, seed, mix in [("HRHN hospitals 2-3", 11, {"code": 300, "appliance": 180, "embedded": 120, "rest": 100}),
("Marrow (frozen rules)", 22, {"code": 500, "appliance": 60, "embedded": 200, "rest": 140})]:
g, lo, hi = gain_ci(corpus(seed, mix))
print(f"{name:24s} recall gain = {g:.3f} 95% CI [{lo:.3f}, {hi:.3f}]")
Expected output:
HRHN hospitals 2-3 recall gain = 0.207 95% CI [0.176, 0.236]
Marrow (frozen rules) recall gain = 0.097 95% CI [0.079, 0.116]
Under these synthetic assumptions, H1 (gain ≥ 0.10 on HRHN) is supported, and H4 (gain ≥ 0.05 on Marrow) is supported, with the smaller Marrow gain predicted by its class mix. The bootstrap interval reflects sampling only; it says nothing about adjudication error or about whether Marrow resembles any real company.
The lab draws per-class detections from the same arbitrary probabilities used in Lab 3 and applies them to two class mixes. The HRHN mix of 700 uses matches the held-out corpus size; Marrow's mix has fewer appliances and more embedded devices. The paired bootstrap resamples items, keeping each item's static and dual outcomes together, which is why the intervals are narrow.
5.6 Worked example: the honest results paragraph
"On the held-out HRHN corpus (synthetic; 700 adjudicated quantum-vulnerable uses), dual-source inventory increased recall by 0.21 (95% bootstrap CI 0.18–0.24) over a competently configured static baseline; precision was 0.93. The gain was concentrated in vendor appliances (0.63) and absent for data at rest (0.00), as predicted (H3). With rules frozen, the gain on an independent corpus was 0.10 (0.08–0.12). Nine and fourteen per cent of assets, respectively, could not be adjudicated and are excluded from the recall denominators; under a worst-case assumption that all unknown assets hold undiscovered vulnerable uses, recall falls by up to 0.12. These results describe two synthetic organizations and a model of detection; they are not estimates for real enterprises." Note what the paragraph refuses to say: it does not claim either organization is "quantum-safe" or that dual-source inventory is sufficient.
5.7 Common errors and limits
- Comparing unpaired recall estimates and reporting overlapping intervals as "no difference".
- Reporting only sampling uncertainty.
- Touching the validation corpus before freezing rules — even to "check the format".
- Dropping unknown assets from the denominator without saying so.
- Lab 5 simulates detection probabilities; real asset classes are messier.
Each error has a concrete guard in this study: the paired design, the three-uncertainty report, the frozen and hashed configuration before Marrow was opened, and the unknown shares of nine and fourteen per cent printed beside recall. A reviewer should be able to confirm all four from the archive without asking the candidate.
Exercises — chapter 5
- EX027 — Why pair? Explain with Lab 5 why the paired interval is narrower than two independent intervals. Rubric: 3 = variance argument correct.
- EX028 — Disputed items. Given 40 disputed items on HRHN, describe the sensitivity analysis and how you would report it. Rubric: 3 = both-ways bounds and reporting sentence.
- EX029 — Specification curve. List four defensible alternative specifications for the main estimate. Rubric: 3 = each alters one analytic choice and is justified.
- EX030 — Freeze protocol. Write the freeze procedure for Marrow, including what is hashed and who witnesses it. Rubric: 3 = verifiable by an outsider.
- EX031 — Predict Marrow. Using Lab 3's per-class probabilities, predict Marrow's aggregate gain from its mix before running Lab 5. Rubric: 3 = prediction within 0.03 and method shown.
- EX032 — Results paragraph. Rewrite an overclaiming paragraph ("Our method makes organizations quantum-safe") in the style of 5.6. Rubric: 3 = scope, uncertainty and unknowns all present.
Chapter synthesis
The main study is a paired comparison with three kinds of uncertainty reported, validated on an untouched corpus under frozen rules, adjudicated blind, and written up with its unknowns in the denominator conversation. A predicted reduction in benefit on a different corpus supports the mechanism.
Continue studying — chapter 05 of 08
← Chapter 04 (draft) · Course chapter index · Chapter 06 (draft) →