OpenCyber AcademyLesson map
Lesson 01 published. Map, example, EX001.
flowchart TD
N0["Learning outcomes"]
N1["1.1 A falsifiable question"]
N2["1.2 Why click rates mislead"]
N3["1.3 Integrity and ethics"]
N4["1.4 Coding reported emails"]
N5["Lab — Coder agreement on reported emails"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX001"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["1.1 A falsifiable question"]
N2["1.2 Why click rates mislead"]
N3["1.3 Integrity and ethics"]
N4["1.4 Coding reported emails"]
N5["Lab — Coder agreement on reported emails"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX001"]
[Learning outcomes]
|
v
[1.1 A falsifiable question]
|
v
[1.2 Why click rates mislead]
|
v
[1.3 Integrity and ethics]
|
v
[1.4 Coding reported emails]
|
v
[Lab — Coder agreement on reported emails]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX001]
Learning outcomes
You will (1) frame a falsifiable question about security behaviour; (2) set integrity and ethics rules for research with employees; (3) measure agreement when coding reported emails.
1.1 A falsifiable question
Mireille Okonjo is a doctoral candidate at the fictional Calder University, supervised by Prof. Henrik Aalto. Her partner is the fictional Vantor Municipal Services, a city council with four thousand staff. Vantor's awareness programme is judged by the share of staff who click simulated phishing links. Mireille's question is narrower and testable: does a programme built on easy reporting and immediate, supportive feedback raise the share of real suspicious emails reported within thirty minutes, compared with the existing click-focused programme? She writes down in advance which result would show her idea is wrong.
1.2 Why click rates mislead
Mireille reviews the critique of click rates as an outcome. Click rates depend heavily on how hard the lure is, so they change when the simulation changes even if staff behaviour does not. Lain, Kostiainen and Čapkun (2022) studied phishing in a large organisation over a long period and found that some popular training practices did not help as expected, while staff reporting was a valuable signal. Dawkins and Jacobs (2023) describe the NIST Phish Scale for rating lure difficulty. She decides to fix lure difficulty in her study and use reporting as the main outcome.
1.3 Integrity and ethics
Research with employees carries risks of deception, embarrassment and power imbalance. Mireille's protocol, approved by Calder's ethics board and Vantor's works council, sets rules: simulations are announced at the organisational level in advance, without dates; no individual results go to managers; nobody is disciplined for clicking; no real credentials are collected, since the landing page records only that a link was opened; and staff who click see a short supportive explanation. She pre-registers hypotheses, outcomes and analysis before collecting data, so she cannot adjust them after seeing results.
1.4 Coding reported emails
Her outcome depends on judging whether a reported real email was actually suspicious. Two trained coders independently label a pilot sample of reports as suspicious or benign using a written guide. The lab computes agreement and Cohen's kappa, which corrects for agreement expected by chance; Landis and Koch (1977) offer conventional bands for interpreting it. Agreement is high enough to proceed, and disagreements lead to a clearer rule about newsletters with tracking links. Measured coder agreement is part of the evidence that her outcome means something stable.
Lab — Coder agreement on reported emails
# Cohen's kappa for two coders labelling 50 pilot reports suspicious (1) or benign (0), synthetic.
a = [1] * 22 + [0] * 3 + [1] * 2 + [0] * 23
b = [1] * 22 + [1] * 3 + [0] * 2 + [0] * 23
n = len(a); po = sum(x == y for x, y in zip(a, b)) / n
pa, pb = sum(a) / n, sum(b) / n
pe = pa * pb + (1 - pa) * (1 - pb)
print(f"reports {n}; agreement {po:.2f}; chance {pe:.3f}; kappa {(po - pe) / (1 - pe):.2f}")
Run with python3 lab_kappa.py. Verified output:
reports 50; agreement 0.90; chance 0.500; kappa 0.80
Worked example
Mireille frames a falsifiable reporting question, replaces click rate with reporting at fixed lure difficulty, writes ethical rules for simulations and measures coder agreement.
Synthetic case. Calder, Vantor and all people are synthetic; Lain et al. (2022), Dawkins and Jacobs (2023) and Landis and Koch (1977) are real.
Errors and limits
Pre-registration does not stop all bias. Kappa depends on prevalence in the sample.
Exercises
- EX001 — Falsifiable outcome. What makes Mireille's question falsifiable?
Show marking criteria
Rubric: 3 = she states in advance which result would show her idea is wrong;; 2 = prior statement;; 1 = vague;; 0 = none. - EX002 — Click-rate flaw. Why are click rates a weak outcome?
Show marking criteria
Rubric: 3 = they depend heavily on lure difficulty;; 2 = difficulty;; 1 = vague;; 0 = none. - EX003 — Lure difficulty tool. Which published tool helps rate lure difficulty?
Show marking criteria
Rubric: 3 = the NIST Phish Scale (Dawkins and Jacobs, 2023);; 2 = Phish Scale;; 1 = vague;; 0 = none. - EX004 — Landing page rule. How does the protocol avoid collecting credentials?
Show marking criteria
Rubric: 3 = the landing page records only that a link was opened;; 2 = no capture;; 1 = vague;; 0 = none. - EX005 — Manager reports. Who receives individual simulation results?
Show marking criteria
Rubric: 3 = nobody; only aggregates are reported;; 2 = aggregates;; 1 = managers;; 0 = none. - EX006 — Kappa result. What kappa did the pilot coding reach in the lab?
Show marking criteria
Rubric: 3 = 0.80 on 50 reports, agreement 0.90;; 2 = about 0.8;; 1 = vague;; 0 = none. - EX007 — Coding rule change. What rule changed after coder disagreements?
Show marking criteria
Rubric: 3 = newsletters with tracking links were defined more clearly;; 2 = newsletters;; 1 = vague;; 0 = none.
Synthesis
Doctoral work on security behaviour needs a falsifiable outcome, strict ethics for simulations and measured agreement on how outcomes are coded.