## Chapter 1 — Advanced research integrity and methods ### Lesson map Lesson 01 published. Map, example, EX001. ```mermaid flowchart TD N0["Learning outcomes"] N1["1.1 A falsifiable question"] N2["1.2 Why click rates mislead"] N3["1.3 Integrity and ethics"] N4["1.4 Coding reported emails"] N5["Lab — Coder agreement on reported emails"] N6["Synthesis"] N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 N6 --> EX["Worked example"] EX --> Q["Exercise EX001"] ``` ### Logical drawing ```mermaid flowchart LR N0["Learning outcomes"] N1["1.1 A falsifiable question"] N2["1.2 Why click rates mislead"] N3["1.3 Integrity and ethics"] N4["1.4 Coding reported emails"] N5["Lab — Coder agreement on reported emails"] N6["Synthesis"] N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX001"] ``` ```text [Learning outcomes] | v [1.1 A falsifiable question] | v [1.2 Why click rates mislead] | v [1.3 Integrity and ethics] | v [1.4 Coding reported emails] | v [Lab — Coder agreement on reported emails] | v [Synthesis] | v [Worked example] --> [Exercise EX001] ``` ### Learning outcomes You will (1) frame a falsifiable question about security behaviour; (2) set integrity and ethics rules for research with employees; (3) measure agreement when coding reported emails. ### 1.1 A falsifiable question Mireille Okonjo is a doctoral candidate at the fictional Calder University, supervised by Prof. Henrik Aalto. Her partner is the fictional Vantor Municipal Services, a city council with four thousand staff. Vantor's awareness programme is judged by the share of staff who click simulated phishing links. Mireille's question is narrower and testable: does a programme built on easy reporting and immediate, supportive feedback raise the share of real suspicious emails reported within thirty minutes, compared with the existing click-focused programme? She writes down in advance which result would show her idea is wrong. ### 1.2 Why click rates mislead Mireille reviews the critique of click rates as an outcome. Click rates depend heavily on how hard the lure is, so they change when the simulation changes even if staff behaviour does not. Lain, Kostiainen and Čapkun (2022) studied phishing in a large organisation over a long period and found that some popular training practices did not help as expected, while staff reporting was a valuable signal. Dawkins and Jacobs (2023) describe the NIST Phish Scale for rating lure difficulty. She decides to fix lure difficulty in her study and use reporting as the main outcome. ### 1.3 Integrity and ethics Research with employees carries risks of deception, embarrassment and power imbalance. Mireille's protocol, approved by Calder's ethics board and Vantor's works council, sets rules: simulations are announced at the organisational level in advance, without dates; no individual results go to managers; nobody is disciplined for clicking; no real credentials are collected, since the landing page records only that a link was opened; and staff who click see a short supportive explanation. She pre-registers hypotheses, outcomes and analysis before collecting data, so she cannot adjust them after seeing results. ### 1.4 Coding reported emails Her outcome depends on judging whether a reported real email was actually suspicious. Two trained coders independently label a pilot sample of reports as suspicious or benign using a written guide. The lab computes agreement and Cohen's kappa, which corrects for agreement expected by chance; Landis and Koch (1977) offer conventional bands for interpreting it. Agreement is high enough to proceed, and disagreements lead to a clearer rule about newsletters with tracking links. Measured coder agreement is part of the evidence that her outcome means something stable. ### Lab — Coder agreement on reported emails ```python # Cohen's kappa for two coders labelling 50 pilot reports suspicious (1) or benign (0), synthetic. a = [1] * 22 + [0] * 3 + [1] * 2 + [0] * 23 b = [1] * 22 + [1] * 3 + [0] * 2 + [0] * 23 n = len(a); po = sum(x == y for x, y in zip(a, b)) / n pa, pb = sum(a) / n, sum(b) / n pe = pa * pb + (1 - pa) * (1 - pb) print(f"reports {n}; agreement {po:.2f}; chance {pe:.3f}; kappa {(po - pe) / (1 - pe):.2f}") ``` Run with `python3 lab_kappa.py`. Verified output: ```text reports 50; agreement 0.90; chance 0.500; kappa 0.80 ``` ### Worked example Mireille frames a falsifiable reporting question, replaces click rate with reporting at fixed lure difficulty, writes ethical rules for simulations and measures coder agreement. > **Synthetic case.** Calder, Vantor and all people are synthetic; Lain et al. (2022), Dawkins and Jacobs (2023) and Landis and Koch (1977) are real. ### Errors and limits Pre-registration does not stop all bias. Kappa depends on prevalence in the sample. ### Exercises - **EX001 — Falsifiable outcome.** What makes Mireille's question falsifiable? *Rubric:* 3 = she states in advance which result would show her idea is wrong;; 2 = prior statement;; 1 = vague;; 0 = none. - **EX002 — Click-rate flaw.** Why are click rates a weak outcome? *Rubric:* 3 = they depend heavily on lure difficulty;; 2 = difficulty;; 1 = vague;; 0 = none. - **EX003 — Lure difficulty tool.** Which published tool helps rate lure difficulty? *Rubric:* 3 = the NIST Phish Scale (Dawkins and Jacobs, 2023);; 2 = Phish Scale;; 1 = vague;; 0 = none. - **EX004 — Landing page rule.** How does the protocol avoid collecting credentials? *Rubric:* 3 = the landing page records only that a link was opened;; 2 = no capture;; 1 = vague;; 0 = none. - **EX005 — Manager reports.** Who receives individual simulation results? *Rubric:* 3 = nobody; only aggregates are reported;; 2 = aggregates;; 1 = managers;; 0 = none. - **EX006 — Kappa result.** What kappa did the pilot coding reach in the lab? *Rubric:* 3 = 0.80 on 50 reports, agreement 0.90;; 2 = about 0.8;; 1 = vague;; 0 = none. - **EX007 — Coding rule change.** What rule changed after coder disagreements? *Rubric:* 3 = newsletters with tracking links were defined more clearly;; 2 = newsletters;; 1 = vague;; 0 = none. ### Synthesis Doctoral work on security behaviour needs a falsifiable outcome, strict ethics for simulations and measured agreement on how outcomes are coded. ### Continue studying — chapter 01 of 08 [← Course home](../../README.md) · [Course chapter index](../SUBJECTS.md) · [Chapter 02 (draft) →](#ch-02) ## Worked answers **EX001.** She writes down beforehand the result that would show reporting-focused training does not help. **EX002.** Because they change with how hard the lure is, even when behaviour does not change. **EX003.** The NIST Phish Scale described by Dawkins and Jacobs (2023). **EX004.** The landing page records only that a link was opened and collects no credentials. **EX005.** Nobody; managers see only aggregate results. **EX006.** Kappa 0.80 with raw agreement 0.90 on 50 pilot reports. **EX007.** A clearer rule for newsletters containing tracking links.