Chapter 5 — Main study and independent validation
Lesson map
Lesson 05 published. Map, example, EX027.
flowchart TD
N0["Learning outcomes"]
N1["5.1 Randomisation and conduct"]
N2["5.2 Primary analysis"]
N3["Lab — Permutation test"]
N4["5.3 Reading the result"]
N5["5.4 Independent validation"]
N6["5.5 Threats that remain"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX027"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["5.1 Randomisation and conduct"]
N2["5.2 Primary analysis"]
N3["Lab — Permutation test"]
N4["5.3 Reading the result"]
N5["5.4 Independent validation"]
N6["5.5 Threats that remain"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX027"]
[Learning outcomes]
|
v
[5.1 Randomisation and conduct]
|
v
[5.2 Primary analysis]
|
v
[Lab — Permutation test]
|
v
[5.3 Reading the result]
|
v
[5.4 Independent validation]
|
v
[5.5 Threats that remain]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX027]
Learning outcomes
You will (1) run a cluster-randomised field study with blinded outcome confirmation; (2) analyse it with the pre-registered permutation test; (3) report primary and secondary results without overstating them.
5.1 Randomisation and conduct
Meridian's security lead, not Nadia, runs the randomisation script with a seed recorded in the registration, assigning six tokenisation-or-webhook teams and six storage-or-platform teams to each arm. The study runs one quarter. Checklist teams complete the form for every crypto-touching change and run the lint step in continuous integration; usual-review teams change nothing. Two auditors who never see arm labels confirm findings weekly from pseudonymised diffs. Nadia monitors only procedural metrics during the quarter, such as checklist completion, so that she cannot be accused of peeking at outcomes.
5.2 Primary analysis
At quarter end each team has a count of confirmed findings per ten merged crypto-touching changes. The observed difference in means is the statistic. Under the null hypothesis that the arm makes no difference, team labels are exchangeable, so the permutation test reshuffles the 24 team values into two groups of twelve many times and asks how often a difference at least as large as the observed one appears by chance. The lab reproduces the registered analysis with 20,000 shuffles and the seed fixed at 2026.
Lab — Permutation test
"""Permutation test: misuse per 10 merged changes, checklist+lint vs usual review (synthetic teams)."""
import random
CHECK = [0, 1, 0, 0, 2, 0, 1, 0, 0, 1, 0, 0]
USUAL = [2, 1, 3, 0, 2, 1, 2, 3, 1, 0, 2, 2]
def mean(x): return sum(x) / len(x)
def main():
obs = mean(USUAL) - mean(CHECK)
pool = CHECK + USUAL; rng = random.Random(2026); n = 20000; hits = 0
for _ in range(n):
rng.shuffle(pool)
if mean(pool[12:]) - mean(pool[:12]) >= obs: hits += 1
print(f"checklist mean {mean(CHECK):.2f}, usual mean {mean(USUAL):.2f}, diff {obs:.2f}")
print(f"one-sided permutation p = {hits / n:.4f} (seed 2026, {n} shuffles)")
print(f"zero-misuse teams: checklist {CHECK.count(0)}/12, usual {USUAL.count(0)}/12")
if __name__ == "__main__":
main()
Run with python3 lab_trial.py. Verified output:
checklist mean 0.42, usual mean 1.58, diff 1.17
one-sided permutation p = 0.0034 (seed 2026, 20000 shuffles)
zero-misuse teams: checklist 8/12, usual 2/12
5.3 Reading the result
Checklist teams average 0.42 confirmed findings per ten changes and usual-review teams 1.58, a difference of 1.17. Only about 0.34 percent of shuffles produce a difference that large, a one-sided p of 0.0034, so H1 is supported at the registered threshold of 0.05. The secondary outcome points the same way: eight of twelve checklist teams record zero misuse against two of twelve controls. Nadia reports both the effect size and the p-value, and she reports the raw team counts in an appendix so readers can recompute everything without trusting her summary.
5.4 Independent validation
Prof. Brandt arranges for a third auditor from another university, under the same data agreement, to re-confirm a random 30 percent of findings blind to arm. Agreement with the original auditors is high, and the re-confirmation changes no team's count by more than one, which leaves the conclusion intact. Nadia also checks the lint-only explanation from Chapter 3: lint warnings explain about a third of avoided nonce findings, but most avoided key-handling findings came from checklist questions the tool cannot see, such as whether a key is issued by the managed key service.
5.5 Threats that remain
Nadia lists remaining threats plainly. Reviewers in checklist teams knew they were observed, a Hawthorne effect that could fade once the study ends. Twelve clusters per arm make the estimate imprecise, so the true effect may be much smaller than 1.17. Auditors could not see secrets injected at deploy time, so key-handling findings remain an under-count in both arms. Finally, Meridian's mature wrapper may make checklists easier to apply than in firms that call low-level primitives directly, which limits transfer to those settings.
Worked example
The registered permutation test gives 0.42 versus 1.58, a difference of 1.17 with p = 0.0034; a blinded third auditor and a lint-only check leave the conclusion intact, with four stated threats.
Synthetic note. All team counts and auditors are invented; the permutation method is standard.
Errors and limits
A small p-value does not measure how large or important an effect is, and a one-quarter study cannot show durability. Never drop teams after unblinding.
Exercises
- EX027 — Who randomised. Who ran the randomisation, and why not Nadia? Rubric: 3 = Meridian's security lead with the registered seed, so the researcher cannot influence allocation;; 2 = who only;; 1 = Nadia;; 0 = none.
- EX028 — Exchangeability logic. Why may team labels be shuffled under the null? Rubric: 3 = if the arm has no effect, labels are exchangeable, so shuffles show chance differences;; 2 = partial;; 1 = "random";; 0 = none.
- EX029 — Main numbers. Report the two means, the difference and the p-value. Rubric: 3 = 0.42, 1.58, 1.17, p = 0.0034;; 2 = three of four;; 1 = one;; 0 = none.
- EX030 — Zero-misuse comparison. What did the secondary outcome show? Rubric: 3 = 8 of 12 checklist teams vs 2 of 12 controls with zero misuse;; 2 = direction;; 1 = wrong counts;; 0 = none.
- EX031 — Lint-only check. What did the lint-only analysis find? Rubric: 3 = lint explains about a third of avoided nonce findings; most key-handling gains came from checklist questions;; 2 = partial;; 1 = lint explains all;; 0 = none.
- EX032 — Remaining threats. Name three remaining threats. Rubric: 3 = three of Hawthorne, imprecision, deploy-time secrets unseen, limited transfer;; 2 = two;; 1 = one;; 0 = none.
Synthesis
A pre-registered, blinded, cluster-randomised test supports the checklist effect; independent re-confirmation and honest threats keep the claim proportionate.
Continue studying — chapter 05 of 08
← Chapter 04 (draft) · Course chapter index · Chapter 06 (draft) →