Chapter 6 — Data collection and analysis
Lesson map
Lesson 06 published. Map, example, EX033.
flowchart TD
N0["Learning outcomes"]
N1["6.1 Collecting the data"]
N2["6.2 The estimate"]
N3["Lab — Difference-in-differences of phishing-simulati"]
N4["6.3 Checking rivals"]
N5["6.4 What the estimate means"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX033"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["6.1 Collecting the data"]
N2["6.2 The estimate"]
N3["Lab — Difference-in-differences of phishing-simulati"]
N4["6.3 Checking rivals"]
N5["6.4 What the estimate means"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX033"]
[Learning outcomes]
|
v
[6.1 Collecting the data]
|
v
[6.2 The estimate]
|
v
[Lab — Difference-in-differences of phishing-simulati]
|
v
[6.3 Checking rivals]
|
v
[6.4 What the estimate means]
|
v
[Worked example] --> [Exercise EX033]
Learning outcomes
You will (1) collect outcome data under a frozen plan; (2) estimate a difference-in-differences effect with a bootstrap interval; (3) check rival explanations against the data.
6.1 Collecting the data
Priya receives pseudonymised human click records for a quarter before and a quarter after feedback starts, taking samples of six hundred staff from each group in each period. The security team applies the two-second rule before export, and Priya checks a random selection of records to confirm it. She also receives department-level report counts, turnover figures and the manager reminder logs. Data are stored on an encrypted university drive, as the ethics approval requires. Priya logs the date of every extract she receives from the team.
6.2 The estimate
The lab shows click rates in feedback departments falling from 15.8 to 6.7 percent, while comparison departments fall from 14.0 to 11.0 percent. The difference-in-differences is minus 6.2 percentage points, with a 95 percent bootstrap interval from minus 10.7 to minus 1.2. The interval excludes zero, so the data support a reduction larger than the shared change. The interval is wide, however, meaning the true effect could be as small as about one percentage point. Priya reports the point estimate and both interval limits together, so readers see the uncertainty alongside the effect. She does not round the upper limit away from zero.
Lab — Difference-in-differences of phishing-simulation click rates with a bootstrap interval
"""Difference-in-differences of phishing-simulation click rates with a bootstrap interval (stdlib only)."""
import random
random.seed(17)
def grp(n, p): return [1 if random.random() < p else 0 for _ in range(n)]
t_pre, t_post = grp(600, 0.16), grp(600, 0.08) # departments with just-in-time feedback
c_pre, c_post = grp(600, 0.15), grp(600, 0.13) # departments with annual training only
m = lambda x: sum(x) / len(x)
did = (m(t_post) - m(t_pre)) - (m(c_post) - m(c_pre))
bs = []
for _ in range(1000):
r = lambda x: [random.choice(x) for _ in x]
bs.append((m(r(t_post)) - m(r(t_pre))) - (m(r(c_post)) - m(r(c_pre))))
bs.sort()
print(f"click rates: feedback {m(t_pre):.1%} -> {m(t_post):.1%}; comparison {m(c_pre):.1%} -> {m(c_post):.1%}")
print(f"difference-in-differences {did*100:.1f} points; 95% bootstrap CI {bs[25]*100:.1f} to {bs[974]*100:.1f}")
Run with python3 lab_did.py. Verified output:
click rates: feedback 15.8% -> 6.7%; comparison 14.0% -> 11.0%
difference-in-differences -6.2 points; 95% bootstrap CI -10.7 to -1.2
6.3 Checking rivals
Templates varied, and the fall appears on new templates as well as familiar ones, weakening the template-learning rival. Filtering rules were identical. Turnover was similar in both groups. Manager reminder logs show slightly more messages in feedback departments, so part of the effect could reflect attention rather than feedback, which Priya reports. Report rates did not fall in feedback departments; they rose slightly, so the embarrassment side effect does not appear. Changing the threshold to five seconds leaves conclusions unchanged. She tabulates each check.
6.4 What the estimate means
Priya interprets the result carefully: in this synthetic setting, feedback departments reduced human clicking by about six percentage points more than comparison departments, with considerable uncertainty. She does not claim feedback prevents breaches, because simulation clicks are a proxy for real behaviour, and real attackers adapt. Dr. Bello reminds her that one organisation over two quarters cannot show the effect persists. She writes these points directly beside the estimate rather than in a distant limitations section. Priya also notes that results could differ in larger organisations.
Worked example
Priya estimates a difference-in-differences of minus 6.2 points (interval minus 10.7 to minus 1.2), finds no fall in reporting, notes the manager reminder concern and states the proxy limits beside the estimate.
Synthetic case. Records are synthetic; outputs come from lab_did.py.
Errors and limits
Treating a simulation click rate as a breach rate overstates the finding.
Exercises
- EX033 — Feedback rates. How did feedback departments' click rates change? Rubric: 3 = 15.8 to 6.7 percent;; 2 = partly correct;; 1 = wrong;; 0 = none.
- EX034 — Comparison rates. How did comparison click rates change? Rubric: 3 = 14.0 to 11.0 percent;; 2 = partly correct;; 1 = wrong;; 0 = none.
- EX035 — DiD and interval. What is the estimate and interval? Rubric: 3 = minus 6.2 points, minus 10.7 to minus 1.2;; 2 = partly correct;; 1 = wrong;; 0 = none.
- EX036 — Remaining rival. Which rival is not fully ruled out? Rubric: 3 = manager reminders;; 2 = partly correct;; 1 = other;; 0 = none.
- EX037 — Reporting result. What happened to report rates? Rubric: 3 = rose slightly in feedback departments;; 2 = partly correct;; 1 = fell;; 0 = none.
- EX038 — Proxy limit. Why not claim feedback prevents breaches? Rubric: 3 = simulation clicks are a proxy;; 2 = partly correct;; 1 = vague;; 0 = none.
Continue studying — lesson 06 (chapter 06 of 08)
← Chapter 05 · Course chapter index · Chapter 07 →
Continue to the next authored lesson.