OpenCyber Academy
Doctoral research · Chapter 5 · English
Course code: OCA-190Chapter 05: OCA-190-C005

Chapter 5 — Main study and independent validation

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 05 published. Map, example, EX027.

flowchart TD
  N0["Learning outcomes"]
  N1["5.1 Running the study"]
  N2["5.2 Estimating the effect"]
  N3["Lab — Stepped-design effect with a permutation check"]
  N4["5.3 Reading the result"]
  N5["5.4 Independent validation"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX027"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["5.1 Running the study"]
  N2["5.2 Estimating the effect"]
  N3["Lab — Stepped-design effect with a permutation check"]
  N4["5.3 Reading the result"]
  N5["5.4 Independent validation"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX027"]

[Learning outcomes]
    |
    v
[5.1 Running the study]
    |
    v
[5.2 Estimating the effect]
    |
    v
[Lab — Stepped-design effect with a permutation check]
    |
    v
[5.3 Reading the result]
    |
    v
[5.4 Independent validation]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX027]

Learning outcomes

You will (1) run the stepped main study; (2) estimate the effect with a permutation check; (3) arrange independent validation.

5.1 Running the study

The main study runs for fourteen months. Every two months three departments switch, until all twelve use the reporting-focused programme. Mireille monitors fidelity: is the button available, are thank-you messages prompt, are simulations at the agreed difficulty? Two months in, an email filter upgrade reduces the volume of suspicious emails reaching staff. Because it affects all departments at once and is logged, her design absorbs it. She publishes monthly fidelity notes to the steering group without any individual data. Each note is reviewed by the works council before circulation.

5.2 Estimating the effect

Mireille compares department-months before and after switching, adjusting for month to absorb organisation-wide changes. The lab runs a simplified version on synthetic data: it estimates the change in thirty-minute reporting after switching and then repeats the estimate many times with the switch order shuffled, to see how often chance alone gives an effect as large. Following her pre-registration, she reports both the estimate and the permutation share rather than a single headline number. She keeps the analysis script frozen from the date of pre-registration, and any later change is recorded with its reason so that reviewers can see exactly what was planned and what was added.

Lab — Stepped-design effect with a permutation check

# Stepped rollout: 12 departments, 7 periods; switch order randomised. Synthetic 30-minute reporting rates.
import random
random.seed(11)
order = [1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4]   # period each department switches
base = [0.22, 0.25, 0.27, 0.24, 0.21, 0.26, 0.23, 0.28, 0.25, 0.22, 0.24, 0.26]
month = [0.0, 0.01, -0.02, 0.00, 0.01, -0.01, 0.0]
rates = [[base[d] + month[t] + (0.17 if t >= order[d] else 0) + random.gauss(0, 0.02) for t in range(7)] for d in range(12)]
def effect(o):
    # regress on treatment after removing department and month means (two-way demeaning)
    y = [row[:] for row in rates]; x = [[1.0 if t >= o[d] else 0.0 for t in range(7)] for d in range(12)]
    for m in (y, x):
        for _ in range(50):
            for d in range(12):
                a = sum(m[d]) / 7; m[d] = [v - a for v in m[d]]
            for t in range(7):
                a = sum(m[d][t] for d in range(12)) / 12
                for d in range(12): m[d][t] -= a
    num = sum(x[d][t] * y[d][t] for d in range(12) for t in range(7))
    den = sum(x[d][t] ** 2 for d in range(12) for t in range(7))
    return num / den
est = effect(order); rng = random.Random(5); k = 0; N = 2000
for _ in range(N):
    o = order[:]; rng.shuffle(o)
    if abs(effect(o)) >= abs(est): k += 1
print(f"estimated change in 30-min reporting: {est:+.3f}; permutation share {k / N:.3f}")

Run with python3 lab_stepped.py. Verified output:

estimated change in 30-min reporting: +0.157; permutation share 0.000

5.3 Reading the result

The lab estimates that reporting rose by about sixteen percentage points (+0.157) after switching, and none of 2,000 shuffled orders produces an effect that large. Simulation click rates fell slightly, but less than reporting rose, and the burden survey showed no increase. Mireille states the finding narrowly: in this council, at fixed lure difficulty, the reporting-focused programme was followed by substantially more timely reporting of real suspicious emails. She does not claim fewer breaches, which the study did not measure. She reports the burden result with the same care as the main effect.

5.4 Independent validation

Mireille arranges for a second analyst at another university to receive the anonymised department-month data, the pre-registration and the analysis plan, but not her code. The analyst writes an independent analysis and reaches an estimate within two percentage points of hers. A second council, the fictional Brenholt District, also runs a shorter replication and finds a smaller but positive effect. Independent validation shows that the result does not depend on one analyst's choices or one organisation's peculiarities. Both validation reports are published alongside her own analysis, with differences explained.

Worked example

Mireille runs a fourteen-month stepped study with fidelity checks, estimates a reporting effect with a permutation check and obtains independent reanalysis and replication.

Synthetic case. Councils, data and results are synthetic.

Errors and limits

Department-level outcomes hide individual variation. Replications in other councils may differ in culture.

Exercises

Synthesis

A stepped study with fidelity checks, permutation inference and independent validation supports a narrow, credible claim.

Continue studying — chapter 05 of 08

← Chapter 04 (draft) · Course chapter index · Chapter 06 (draft) →

Worked answers

EX027. That the button was available, thank-yous were prompt and simulations stayed at the agreed difficulty.

EX028. Because it hit all departments at the same time, was logged and was absorbed by month adjustment.

EX029. About sixteen percentage points (+0.157) more timely reporting, with no shuffled order out of 2,000 as large.

EX030. As more timely reporting in this council at fixed lure difficulty, without claiming fewer breaches.

EX031. Anonymised department-month data, the pre-registration and the analysis plan, but not Mireille's code.

EX032. A smaller but still positive effect.