Chapter 15 — Evidence and measurement

Lesson map

Lesson 15 published. Map, example, EX043.

flowchart TD
  N0["Learning outcomes"]
  N1["15.1 Valid measures"]
  N2["15.2 Outputs and outcomes"]
  N3["Lab — Median time to first report"]
  N4["15.3 Measuring respectfully"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX043"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["15.1 Valid measures"]
  N2["15.2 Outputs and outcomes"]
  N3["Lab — Median time to first report"]
  N4["15.3 Measuring respectfully"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX043"]

[Learning outcomes]
    |
    v
[15.1 Valid measures]
    |
    v
[15.2 Outputs and outcomes]
    |
    v
[Lab — Median time to first report]
    |
    v
[15.3 Measuring respectfully]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX043]

Learning outcomes

You will (1) choose valid measures of security behaviour; (2) distinguish outputs from outcomes; (3) measure without harming trust.

15.1 Valid measures

A measure is valid if it captures what it claims to. Course completion rates show that people clicked through a module, not that they behave securely. Simulation click rates depend heavily on how hard the email was, which the NIST Phish Scale (Dawkins and Jacobs, 2023) helps to rate. Nadia lists candidate measures for Ashcombe and judges each for validity. Reporting rates for real suspicious emails and time to first report are closer to what matters: how quickly the organisation learns of an attack.

15.2 Outputs and outcomes

Outputs are things the programme produces, such as videos shown or emails sent. Outcomes are changes in the world, such as fewer successful frauds or faster reporting. Nadia sorts Ashcombe's current figures and finds they are all outputs. She proposes two outcome measures: median time from a malicious email's arrival to the first report, and the number of fraudulent payment requests caught. The lab computes the median time from a synthetic month of data, showing how an outcome can be calculated from existing records.

Lab — Median time to first report

# Median minutes from arrival to first report for a month of malicious emails (synthetic).
times = [4, 7, 9, 11, 12, 14, 15, 22, 30, 41, 65]
s = sorted(times); print("emails:", len(s), "median:", s[len(s) // 2], "min")

Run with python3 lab_median.py. Verified output:

emails: 11 median: 14 min

15.3 Measuring respectfully

Measurement can harm trust if staff feel watched. Nadia proposes that results are reported by branch groups of twenty or more, that individual data are not shown to managers and that staff see the same summary as the board. She also proposes asking staff every few months how much effort security takes. Measuring burden alongside outcomes reminds leaders that security effort has a cost, and that a programme that improves outcomes by exhausting staff is unlikely to last. Staff representatives review the proposal.

Worked example

Nadia judges measures for validity, separates outputs from outcomes, computes a median report time and proposes respectful measurement.

Synthetic case. Data are synthetic; Dawkins and Jacobs (2023) is real.

Errors and limits

Outcome measures can be noisy. Unreported attacks are invisible.

Exercises

• EX043 — Completion validity. Why are completion rates weak measures?
  Show marking criteria

  Rubric: 3 = they show clicking through, not behaviour;; 2 = validity;; 1 = vague;; 0 = none.
• EX044 — Median result. What was the median time to first report in the lab?
  Show marking criteria

  Rubric: 3 = 14 minutes;; 2 = 14;; 1 = vague;; 0 = none.
• EX045 — Burden measure. Why measure staff effort too?
  Show marking criteria

  Rubric: 3 = effort is a real cost that affects sustainability;; 2 = cost;; 1 = vague;; 0 = none.

Synthesis

Valid outcome measures, collected respectfully, show whether a programme helps.

Continue studying — chapter 15 of 40

← Chapter 14 (draft) · Course chapter index · Chapter 16 (draft) →