◇ OpenCyber Academy
Doctoral research · Chapter 5 · English

Chapter 5 — Main study and independent validation

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 5 — Main study and independent validation

Lesson map

Lesson 05 published. Map, example, EX027.

flowchart TD
  N0["Learning outcomes"]
  N1["5.1 Running the study"]
  N2["Lab — Residual comparison"]
  N3["5.2 Results"]
  N4["5.3 Independent validation"]
  N5["5.4 Interpreting carefully"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX027"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["5.1 Running the study"]
  N2["Lab — Residual comparison"]
  N3["5.2 Results"]
  N4["5.3 Independent validation"]
  N5["5.4 Interpreting carefully"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX027"]

[Learning outcomes]
    |
    v
[5.1 Running the study]
    |
    v
[Lab — Residual comparison]
    |
    v
[5.2 Results]
    |
    v
[5.3 Independent validation]
    |
    v
[5.4 Interpreting carefully]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX027]

Learning outcomes

You will (1) run the randomised comparison; (2) analyse residual issues; (3) validate findings independently.

5.1 Running the study

Over eight weeks, partner auditors review twenty-four contracts, twelve with the checklist and twelve with usual review, following the random allocation. Lucía monitors fidelity by checking that checklist reviews record every item and that usual reviews do not borrow the checklist. Each review closes with a findings report. The blinded independent reviewer then examines every contract and records high-severity issues not in the first report. Fidelity checks matter because a comparison is meaningless if the procedures blur together in practice.

Lab — Residual comparison

"""Compare residual high-severity findings after two review procedures (synthetic, read-only).
Residual = high-severity issues found by the blinded independent reviewer after the first review closed."""
import random, statistics
CHECKLIST = [0, 1, 0, 0, 1, 0, 2, 0, 0, 1, 0, 0]
USUAL = [1, 2, 0, 1, 2, 1, 3, 0, 1, 2, 1, 1]

def main():
    m1, m2 = statistics.mean(CHECKLIST), statistics.mean(USUAL)
    diff = m2 - m1
    print(f"contracts per arm {len(CHECKLIST)}")
    print(f"mean residual high: checklist {m1:.2f}, usual {m2:.2f}, difference {diff:.2f}")
    rng = random.Random(2026)
    pool = CHECKLIST + USUAL
    hits, reps = 0, 10000
    for _ in range(reps):
        rng.shuffle(pool)
        if statistics.mean(pool[12:]) - statistics.mean(pool[:12]) >= diff:
            hits += 1
    print(f"one-sided permutation p = {hits / reps:.4f} ({reps} shuffles, seed 2026)")
    print(f"contracts with zero residual high: checklist {CHECKLIST.count(0)}/12, usual {USUAL.count(0)}/12")

if __name__ == "__main__":
    main()

Run with python3 lab_trial.py. Verified output:

contracts per arm 12
mean residual high: checklist 0.42, usual 1.25, difference 0.83
one-sided permutation p = 0.0146 (10000 shuffles, seed 2026)
contracts with zero residual high: checklist 8/12, usual 2/12

5.2 Results

The run reports mean residual high-severity issues of 0.42 per contract with the checklist and 1.25 with usual review, a difference of 0.83. The one-sided permutation test with seed 2026 and ten thousand shuffles gives p equal to 0.0146. Eight of twelve checklist contracts have no residual high issues, compared with two of twelve under usual review. Lucía reports the difference with its uncertainty and notes that twelve contracts per arm is a small sample, so the estimate is imprecise.

A p-value of 0.0146 makes chance an unlikely explanation for the 0.83 difference, but it says nothing about how large the true difference is.

5.3 Independent validation

To check that residual issues are real, a second independent reviewer, also blinded, assesses each residual issue for validity and severity. Agreement on severity is high, and two residual issues from the usual arm are downgraded to medium, which Lucía reports along with a sensitivity analysis that excludes them. The direction of the result holds. Validation guards against the possibility that one reviewer's judgements, rather than the procedures, produced the difference. Lucía records the validator's reasoning for each changed rating, and reports both the original and validated results side by side.

5.4 Interpreting carefully

Lucía writes that structured checklist review was associated with fewer residual high-severity issues in this sample of synthetic contracts and partner auditors. She does not claim the checklist prevents all serious issues, nor that the effect size will hold for public decentralised finance protocols with complex economic designs. She also notes that checklist reviews took on average fifteen percent longer, a cost partners must weigh. Careful interpretation keeps the claim within what the design supports. She states the limits in the abstract as well as the discussion.

Worked example

Residual high-severity issues average 0.42 with the checklist and 1.25 with usual review (p = 0.0146, permutation); 8 of 12 checklist contracts have none, against 2 of 12.

Synthetic case. Contracts, auditors and figures are synthetic.

Errors and limits

Reporting only a p-value without effect size, sample size and limits overstates certainty.

Exercises

Synthesis

A randomised, blinded comparison with independent validation supports a careful, bounded claim.

Continue studying — chapter 05 of 08

← Chapter 04 (draft) · Course chapter index · Chapter 06 (draft) →