Chapter 16 — Human factors and cyber policy: methods II

Lesson map

Lesson 16 published. Map, example, EX046.

flowchart TD
  N0["Learning outcomes"]
  N1["16.1 Before and after"]
  N2["16.2 Staggered rollout"]
  N3["Lab — Early and late clinics during the staggered fo"]
  N4["16.3 Interpreting carefully"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX046"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["16.1 Before and after"]
  N2["16.2 Staggered rollout"]
  N3["Lab — Early and late clinics during the staggered fo"]
  N4["16.3 Interpreting carefully"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX046"]

[Learning outcomes]
    |
    v
[16.1 Before and after]
    |
    v
[16.2 Staggered rollout]
    |
    v
[Lab — Early and late clinics during the staggered fo]
    |
    v
[16.3 Interpreting carefully]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX046]

Learning outcomes

You will (1) compare reporting before and after a change; (2) consider other explanations; (3) use a staggered rollout as a comparison.

16.1 Before and after

After the button and same-day replies are introduced, Lucas compares reporting over two months before and two months after. Reports of real malicious emails rose and the median time to first report fell. He is careful: other factors may have changed, such as the number of attacks or news coverage of a dental practice breach. He checks the IT contractor's filtering logs and finds that attack volume was similar in both periods, which makes the change more convincing. He records both checks in his report.

16.2 Staggered rollout

Because of a delay, four clinics received the button two weeks after the others. Lucas uses those two weeks as a comparison. During that period, clinics with the button reported a higher share of malicious emails than clinics without it. The lab computes both shares. He notes that clinics were not chosen randomly, as the delay was due to older computers, so the groups may differ. Still, the comparison adds weight to the before and after result. He records the reason for the delay.

Lab — Early and late clinics during the staggered fortnight

# Staggered fortnight: share of malicious emails reported (synthetic).
early = (8, 10); late = (4, 9)
for n, (r, t) in (("early clinics", early), ("late clinics", late)): print(f"{n}: {r}/{t} = {r / t:.0%}")

Run with python3 lab_stagger.py. Verified output:

early clinics: 8/10 = 80%
late clinics: 4/9 = 44%

16.3 Interpreting carefully

Lucas writes that the evidence is consistent with the button and replies improving reporting, supported by stable attack volume and the staggered comparison, but not proven. He recommends continuing to monitor. Marta praises the habit of seeking a comparison wherever possible, noting that many workplace changes can be evaluated more convincingly at almost no cost if someone notices a natural comparison and records it carefully. He also lists what further data would strengthen the conclusion, such as a longer period of monitoring and figures from clinics that joined later.

Worked example

Lucas compares before and after with stable attack volume and uses a delayed rollout as a comparison.

Synthetic case. Data are synthetic.

Errors and limits

Non-random groups. Short periods.

Exercises

• EX046 — Volume check. How did Lucas check for changing attack volume?
  Show marking criteria

  Rubric: 3 = filtering logs showed similar volume;; 2 = logs;; 1 = vague;; 0 = none.
• EX047 — Lab shares. What shares did early and late clinics report in the lab?
  Show marking criteria

  Rubric: 3 = 80% versus 44%;; 2 = early higher;; 1 = vague;; 0 = none.
• EX048 — Late cause. Why were some clinics late?
  Show marking criteria

  Rubric: 3 = older computers;; 2 = hardware;; 1 = random;; 0 = none.

Synthesis

Natural comparisons strengthen before and after evaluations at little cost.

Continue studying — chapter 16 of 30

← Chapter 15 (draft) · Course chapter index · Chapter 17 (draft) →