Chapter 6 — Human factors and cyber policy: controlled evaluation
Lesson map
Lesson 06 published. Map, example, EX026.
flowchart TD
N0["Learning outcomes"]
N1["6.1 Designing the evaluation"]
N2["6.2 Within-month analysis"]
N3["Lab — Stepped rollout analysis with permutation test"]
N4["6.3 Ethical constraints"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX026"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["6.1 Designing the evaluation"]
N2["6.2 Within-month analysis"]
N3["Lab — Stepped rollout analysis with permutation test"]
N4["6.3 Ethical constraints"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX026"]
[Learning outcomes]
|
v
[6.1 Designing the evaluation]
|
v
[6.2 Within-month analysis]
|
v
[Lab — Stepped rollout analysis with permutation test]
|
v
[6.3 Ethical constraints]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX026]
Learning outcomes
You will (1) design a controlled evaluation of a programme change; (2) analyse results with appropriate tests; (3) address ethical constraints in evaluation.
6.1 Designing the evaluation
Daniel plans to evaluate the new feedback messages that tell reporters within an hour whether their report was malicious. He uses a stepped rollout: six offices start in month one, six in month two and six in month three, with order chosen randomly. Every office eventually receives the change, which addresses fairness concerns. The outcome is the share of real malicious emails reported by at least one person in each office per month, using data that the reporting system already collects without extra burden.
6.2 Within-month analysis
The lab compares office-months with and without feedback, adjusting for month by comparing within each month. Offices with feedback show a higher share of malicious emails reported. Daniel computes the average within-month difference and checks it with a permutation test that shuffles which offices had feedback. The observed difference is larger than almost all shuffled differences. He reports the estimate, the permutation result and the limits, including that office sizes differ and that the period is short. The difference is about 14 points.
Lab — Stepped rollout analysis with permutation test
# Stepped rollout: 18 offices, start months 1-3 (6 each), 4 months of data (synthetic).
import random
r = random.Random(3)
start = [1] * 6 + [2] * 6 + [3] * 6; r.shuffle(start)
base = [r.uniform(0.45, 0.65) for _ in range(18)]
y = {(o, m): min(1, base[o] + 0.03 * m + (0.16 if m >= start[o] else 0) + r.gauss(0, 0.05)) for o in range(18) for m in range(4)}
def effect(st):
diffs = []
for m in range(1, 4):
t = [y[o, m] for o in range(18) if m >= st[o]]; c = [y[o, m] for o in range(18) if m < st[o]]
if t and c: diffs.append(sum(t) / len(t) - sum(c) / len(c))
return sum(diffs) / len(diffs)
obs = effect(start)
cnt = 0; N = 2000
for _ in range(N):
s = start[:]; r.shuffle(s); cnt += effect(s) >= obs
print(f"within-month difference {100 * obs:.1f} points; permutation share {cnt / N:.3f}")
Run with python3 lab_stepped.py. Verified output:
within-month difference 14.4 points; permutation share 0.001
6.3 Ethical constraints
The evaluation uses existing data in aggregate, so individual consent is not needed for each staff member, but Daniel informs staff through the charter and newsletters that the feedback change is being evaluated by office. The governance panel approves the plan. No simulations are part of the evaluation. He notes that an evaluation that withheld useful feedback indefinitely from some offices would be unfair; the stepped design limits delay to two months, which the panel considers acceptable for the knowledge gained.
Worked example
Daniel evaluates feedback messages with a randomised stepped rollout across eighteen offices, finds a positive within-month difference and documents ethical approval.
Synthetic case. Offices and data are synthetic.
Errors and limits
Short evaluations miss long-term effects. Offices differ in exposure.
Exercises
- EX026 — Rollout order. How was the rollout order chosen? Rubric: 3 = randomly;; 2 = random;; 1 = by size;; 0 = none.
- EX027 — Outcome measure. What outcome was used? Rubric: 3 = share of malicious emails reported per office per month;; 2 = reported share;; 1 = clicks;; 0 = none.
- EX028 — Lab estimate. What within-month difference did the lab find? Rubric: 3 = about 14.4 points, permutation share 0.001;; 2 = positive;; 1 = vague;; 0 = none.
- EX029 — No simulations. Were simulations part of the evaluation? Rubric: 3 = no; it used real reported emails only;; 2 = no, without saying what was used;; 1 = yes;; 0 = no answer.
- EX030 — Maximum delay. What was the maximum delay for any office? Rubric: 3 = two months;; 2 = 2;; 1 = vague;; 0 = none.
Synthesis
Controlled evaluations can be fair, ethical and informative when rollout is staged and randomised.
Continue studying — chapter 06 of 12
← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →