Chapter 3 — Human factors and cyber policy: specialist evidence
Lesson map
Lesson 03 published. Map, example, EX011.
flowchart TD
N0["Learning outcomes"]
N1["3.1 Appraising studies"]
N2["3.2 Effect sizes"]
N3["Lab — Risk difference and interval for two formats"]
N4["3.3 Transferability"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX011"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["3.1 Appraising studies"]
N2["3.2 Effect sizes"]
N3["Lab — Risk difference and interval for two formats"]
N4["3.3 Transferability"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX011"]
[Learning outcomes]
|
v
[3.1 Appraising studies]
|
v
[3.2 Effect sizes]
|
v
[Lab — Risk difference and interval for two formats]
|
v
[3.3 Transferability]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX011]
Learning outcomes
You will (1) appraise specialist studies of security behaviour; (2) interpret effect sizes and uncertainty; (3) judge transferability to an organisation.
3.1 Appraising studies
Daniel appraises key studies in detail. For each he asks what was studied, with whom, how outcomes were measured, whether there was a comparison and what limits the authors state. Lain and colleagues studied thousands of employees in one company over fifteen months with randomised conditions, giving strong internal validity. Adams and Sasse (1999) used questionnaires and interviews to explain password behaviour, offering depth rather than measurement of effects. Appraisal helps him use each study for what it can support. He keeps notes on each one.
3.2 Effect sizes
Daniel learns to read effect sizes, not just significance. A difference may be statistically significant yet small in practice, or large but uncertain. The lab computes a risk difference and its confidence interval for a synthetic comparison of reporting rates between two training formats. The interval is wide, so although one format looks better, the data are consistent with a small or even zero benefit. He notes that decisions should consider the plausible range, not only the point estimate. He reports both numbers.
Lab — Risk difference and interval for two formats
# Risk difference with Wald 95% interval for two training formats (synthetic).
from math import sqrt
a, n1, b, n2 = 37, 100, 31, 100
p1, p2 = a / n1, b / n2; d = p1 - p2; se = sqrt(p1 * (1 - p1) / n1 + p2 * (1 - p2) / n2)
print(f"format A {p1:.0%}, format B {p2:.0%}, difference {100 * d:.1f} points, 95% CI {100 * (d - 1.96 * se):.1f} to {100 * (d + 1.96 * se):.1f}")
Run with python3 lab_effect.py. Verified output:
format A 37%, format B 31%, difference 6.0 points, 95% CI -7.1 to 19.1
3.3 Transferability
A finding from one organisation may not hold in another. Daniel compares Calloway with the organisations in key studies: size, sector, culture and attack exposure. Calloway resembles the large company in Lain's study in its email-heavy work, so the reporting finding likely transfers; it differs from the university settings in some password studies. He records his transferability judgements explicitly, so that others can see why he relies more on some studies than others when designing Calloway's programme. Dr Holm reviews these judgements.
Worked example
Daniel appraises studies by design, reads an effect size with its interval and judges transferability to Calloway.
Synthetic case. Lab data are synthetic; cited studies are real.
Errors and limits
Transferability is a judgement. Published studies favour positive results.
Exercises
- EX011 — Lain design strength. What gives Lain et al. strong internal validity? Rubric: 3 = randomised conditions over fifteen months;; 2 = randomisation;; 1 = vague;; 0 = none.
- EX012 — Adams and Sasse method. What did Adams and Sasse (1999) offer? Rubric: 3 = depth from questionnaires and interviews;; 2 = depth;; 1 = effect size;; 0 = none.
- EX013 — Lab difference. What risk difference did the lab compute? Rubric: 3 = 6.0 points, interval -7.1 to 19.1;; 2 = wide;; 1 = vague;; 0 = none.
- EX014 — Interval meaning. What does the wide interval mean? Rubric: 3 = benefit could be small or zero;; 2 = uncertain;; 1 = vague;; 0 = none.
- EX015 — Transfer judgement. Why does Lain's reporting finding likely transfer? Rubric: 3 = Calloway resembles the large email-heavy company;; 2 = similarity;; 1 = vague;; 0 = none.
Synthesis
Specialist evidence is appraised by design, read with uncertainty and transferred with explicit judgement.
Continue studying — chapter 03 of 12
← Chapter 02 (draft) · Course chapter index · Chapter 04 (draft) →