Chapter 4 — Information security: competing explanations
Lesson map
Lesson 04 published. Map, example, EX021.
flowchart TD
N0["Learning outcomes"]
N1["4.1 Why rivals matter"]
N2["4.2 Designing checks"]
N3["4.3 The reporting explanation"]
N4["4.4 Evidence that would weaken the hypothesis"]
N0 --> N1 --> N2 --> N3 --> N4
N4 --> EX["Worked example"]
EX --> Q["Exercise EX021"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["4.1 Why rivals matter"]
N2["4.2 Designing checks"]
N3["4.3 The reporting explanation"]
N4["4.4 Evidence that would weaken the hypothesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX021"]
[Learning outcomes]
|
v
[4.1 Why rivals matter]
|
v
[4.2 Designing checks]
|
v
[4.3 The reporting explanation]
|
v
[4.4 Evidence that would weaken the hypothesis]
|
v
[Worked example] --> [Exercise EX021]
Learning outcomes
You will (1) list rival explanations for a security improvement; (2) design checks for each rival; (3) state evidence that would weaken your hypothesis.
4.1 Why rivals matter
Priya expects feedback departments to click less, but other causes could produce the same pattern. Staff might learn to recognise simulation emails rather than phishing in general. A mail filter change might block more lures in some departments. Staff turnover might replace experienced people with newcomers in comparison departments. Managers in feedback departments might remind staff more often because they know a pilot is running. Each of these would reduce clicks without feedback being the reason, so each needs a check. Priya lists them.
4.2 Designing checks
For each rival, Priya designs a check. She asks the security team to vary simulation templates so staff cannot learn a single pattern. She confirms with the mail administrators that filtering rules were identical across departments during the study. She compares turnover between groups using anonymised human resources counts. For manager reminders, she asks department heads to log any security messages they send, accepting that this measure depends on honest reporting and noting it as a weakness. Each check is written into the plan before collection.
4.3 The reporting explanation
Peers had suggested that feedback might embarrass staff, making them stop reporting suspicious emails to avoid attention. This would be a harmful side effect, because reports help the security team find real attacks quickly. Priya's secondary outcome, the report rate, tests this directly. If clicks fall but reports also fall, the intervention may be trading one risk for another. She writes this possibility into her plan so a fall in reports cannot be ignored if it appears. The security team supports including this outcome.
4.4 Evidence that would weaken the hypothesis
Priya writes down results that would count against her: a difference-in-differences interval that includes zero, pre-period trends that diverge, a fall in report rates in feedback departments or turnover differences large enough to explain the change. Dr. Bello adds one more: a fall in clicks only on repeated templates and not on new ones. Recording these in advance means she cannot reinterpret disappointing results afterwards, and examiners can check her conclusions against her stated tests. Dr. Bello signs the list, and Priya attaches it to the analysis plan.
Worked example
Priya lists template learning, filter changes, turnover and manager reminders as rivals, designs a check for each, tests the reporting side effect and records weakening results in advance.
Synthetic case. Rivals and checks are synthetic.
Errors and limits
Ignoring a fall in reporting would hide a harmful side effect.
Exercises
- EX021 — Template rival. Why vary simulation templates? Rubric: 3 = so staff cannot learn one pattern;; 2 = partly correct;; 1 = vague;; 0 = none.
- EX022 — Filter check. How is the mail filter rival checked? Rubric: 3 = confirm identical rules across departments;; 2 = partly correct;; 1 = vague;; 0 = none.
- EX023 — Weak measure. Which check depends on honest reporting? Rubric: 3 = manager reminder logs;; 2 = partly correct;; 1 = other;; 0 = none.
- EX024 — Side effect. What harmful side effect might feedback cause? Rubric: 3 = fewer reports of suspicious emails;; 2 = partly correct;; 1 = vague;; 0 = none.
- EX025 — Supervisor addition. What weakening result does Dr. Bello add? Rubric: 3 = fall only on repeated templates;; 2 = partly correct;; 1 = vague;; 0 = none.
- EX026 — Advance purpose. Why record weakening results in advance? Rubric: 3 = prevents reinterpretation afterwards;; 2 = partly correct;; 1 = vague;; 0 = none.
Continue studying — lesson 04 (chapter 04 of 08)
← Chapter 03 · Course chapter index · Chapter 05 →
Continue to the next authored lesson.