Chapter 37 — Evidence review seminar

Lesson map

Lesson 37 published. Map, example, EX109.

flowchart TD
  N0["Learning outcomes"]
  N1["37.1 Reviewing evidence"]
  N2["37.2 Assessing quality"]
  N3["37.3 What is known"]
  N4["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4
  N4 --> EX["Worked example"]
  EX --> Q["Exercise EX109"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["37.1 Reviewing evidence"]
  N2["37.2 Assessing quality"]
  N3["37.3 What is known"]
  N4["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX109"]

[Learning outcomes]
    |
    v
[37.1 Reviewing evidence]
    |
    v
[37.2 Assessing quality]
    |
    v
[37.3 What is known]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX109]

Learning outcomes

You will (1) review evidence on awareness training effectiveness; (2) assess study quality; (3) summarise what is and is not known.

37.1 Reviewing evidence

In a seminar, Nadia and classmates review published evidence on whether awareness training and simulations work. They read Bada, Sasse and Nurse (2019) on why campaigns fail, Lain, Kostiainen and Čapkun (2022) on a large organisational phishing study and NIST SP 800-50r1 on programme design. They note that many vendor claims lack independent evidence. Reviewing evidence helps them avoid repeating popular beliefs and instead base recommendations on studies whose methods they can examine and judge. Each student presents one paper to the group and leads ten minutes of discussion.

37.2 Assessing quality

The group assesses study quality using simple questions: how many participants, was there a comparison group, how were outcomes measured and who funded the study. Lain and colleagues studied thousands of employees over more than a year, which gives their findings weight; they found that embedded training after clicks did not improve outcomes, while employees reported many phishing emails collectively. Small studies with no comparison group or with vendor funding receive less weight. Quality assessment helps them weigh conflicting claims sensibly.

37.3 What is known

The group summarises: evidence suggests that fear-based campaigns and long post-click training are weak; that reporting by many staff can detect attacks quickly; and that usable design reduces burden. Less is known about the long-term effects of feedback or about small organisations like libraries. Nadia notes that her own Ashcombe work adds a small case to this gap. A clear statement of what is known and unknown guides both practice and future research, and protects against overconfident claims. The summary fits on one page.

Worked example

Nadia's seminar reviews key studies, assesses quality and summarises known and unknown points.

Synthetic case. The seminar is synthetic; Bada et al. (2019), Lain et al. (2022) and SP 800-50r1 are real.

Errors and limits

Reviews may miss studies. Quality questions are simple.

Exercises

• EX109 — Lain finding. What did Lain et al. find about embedded training?
  Show marking criteria

  Rubric: 3 = it did not improve outcomes;; 2 = ineffective;; 1 = helped;; 0 = none.
• EX110 — Quality questions. Name two quality questions used.
  Show marking criteria

  Rubric: 3 = number of participants and comparison group;; 2 = two;; 1 = vague;; 0 = none.
• EX111 — Known gap. What is less well known?
  Show marking criteria

  Rubric: 3 = long-term feedback effects and small organisations;; 2 = gap;; 1 = vague;; 0 = none.

Synthesis

Evidence review weighs study quality and states what is and is not known.

Continue studying — chapter 37 of 40

← Chapter 36 (draft) · Course chapter index · Chapter 38 (draft) →