Chapter 26 — Human factors and cyber policy: methods II
Lesson map
Lesson 26 published. Map, example, EX076.
flowchart TD
N0["Learning outcomes"]
N1["26.1 Before and after"]
N2["26.2 Confounding"]
N3["26.3 Comparison groups"]
N4["Lab — Early and late branches in the rollout month"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX076"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["26.1 Before and after"]
N2["26.2 Confounding"]
N3["26.3 Comparison groups"]
N4["Lab — Early and late branches in the rollout month"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX076"]
[Learning outcomes]
|
v
[26.1 Before and after]
|
v
[26.2 Confounding]
|
v
[26.3 Comparison groups]
|
v
[Lab — Early and late branches in the rollout month]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX076]
Learning outcomes
You will (1) analyse change over time with before and after data; (2) recognise confounding factors; (3) use a comparison group where possible.
26.1 Before and after
After Ashcombe adds the report button, states that unsure reports are welcome and promises replies, Nadia compares reporting before and after. In the three months before, staff reported 12 suspicious emails a month on average; in the three months after, 31. She computes the change and notes it is large. Yet a simple before and after comparison cannot tell whether the changes caused the increase, because other things may also have changed during the same period, such as the number of attacks. She plots both periods.
26.2 Confounding
A confounder is something that affects both the timing of a change and the outcome. Nadia considers several: a national phishing campaign against libraries during the after period, news coverage of a library breach and the arrival of new staff. Any of these might have raised reporting. She checks IT records and finds that malicious emails rose too, from 3 to 5 a month, which could explain part of the increase. Considering confounders openly is essential to avoid claiming more than the evidence supports.
26.3 Comparison groups
Nadia learns that a comparison group helps separate effects. Two branches received the button a month later than others because of a technical delay. During that month, reporting at early branches rose sharply while late branches changed little. The lab computes reports per hundred staff for each group in that month. The difference supports the idea that the changes, not only outside events, raised reporting. She notes that the groups were not chosen randomly, so they may differ in other ways too.
Lab — Early and late branches in the rollout month
# Rollout month: reports per hundred staff for early and late branches (synthetic).
early = (25, 96); late = (2, 24)
for name, (r, s) in (("early", early), ("late", late)): print(f"{name}: {r} reports / {s} staff = {100 * r / s:.0f} per 100")
Run with python3 lab_compare.py. Verified output:
early: 25 reports / 96 staff = 26 per 100
late: 2 reports / 24 staff = 8 per 100
Worked example
Nadia compares before and after reporting, considers confounders and uses a delayed rollout as a comparison group.
Synthetic case. Data are synthetic.
Errors and limits
Non-random groups may differ. One month is short.
Exercises
- EX076 — Monthly change. How did average monthly reports change? Rubric: 3 = from 12 to 31;; 2 = rise of 19;; 1 = vague;; 0 = none.
- EX077 — Confounder found. Which confounder did IT records reveal? Rubric: 3 = malicious emails rose from 3 to 5 a month;; 2 = more attacks;; 1 = vague;; 0 = none.
- EX078 — Group rates. What rates did early and late branches show in the rollout month? Rubric: 3 = 26 versus 8 per hundred staff;; 2 = early higher;; 1 = vague;; 0 = none.
Synthesis
Before and after data need confounders considered and comparison groups where possible.
Continue studying — chapter 26 of 40
← Chapter 25 (draft) · Course chapter index · Chapter 27 (draft) →