◇ OpenCyber Academy
Engineering · Chapter 38 · English

Chapter 38 — Human factors and cyber policy: validation

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 38 — Human factors and cyber policy: validation

Lesson map

Lesson 38 published. Map, example, EX112.

flowchart TD
  N0["Learning outcomes"]
  N1["38.1 Validation versus verification"]
  N2["38.2 Validating outcomes"]
  N3["Lab — Before and after reporting outcomes"]
  N4["38.3 Mixed evidence"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX112"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["38.1 Validation versus verification"]
  N2["38.2 Validating outcomes"]
  N3["Lab — Before and after reporting outcomes"]
  N4["38.3 Mixed evidence"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX112"]

[Learning outcomes]
    |
    v
[38.1 Validation versus verification]
    |
    v
[38.2 Validating outcomes]
    |
    v
[Lab — Before and after reporting outcomes]
    |
    v
[38.3 Mixed evidence]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX112]

Learning outcomes

You will (1) distinguish validation from verification; (2) validate the programme against its purpose; (3) use mixed evidence to judge success.

38.1 Validation versus verification

Verification asks whether the system was built as specified; validation asks whether it achieves its purpose. Greyford's reporting service passed verification in chapter 30, but Aiko must now ask whether it actually reduces harm from deception. A service can meet every requirement and still fail its purpose if, for example, staff report only harmless emails while attacks go unreported. Validation therefore looks at outcomes in real operation, not test conditions, and needs evidence collected over months rather than days. Both are needed, in that order.

38.2 Validating outcomes

After nine months, Aiko compares outcomes with the year before launch. The lab computes the change in median time to first report and the share of malicious emails reported. Reporting share rises and median time falls. She also checks payment fraud attempts: three fake bank detail changes were caught by the hold control, against one loss the year before. She notes that the before-and-after comparison cannot rule out changes in attacks, so she presents the findings as consistent with success rather than proof of it.

Lab — Before and after reporting outcomes

# Before and after reporting outcomes for malicious emails (synthetic).
before = {"reported": 41, "total": 100, "times": [12, 20, 25, 30, 38, 45, 60, 75, 90]}
after = {"reported": 78, "total": 100, "times": [3, 4, 6, 8, 9, 11, 14, 18, 25]}
for name, d in (("before", before), ("after", after)):
    t = sorted(d["times"]); print(f"{name}: reported share {d['reported'] / d['total']:.0%}, median first report {t[len(t) // 2]} min")

Run with python3 lab_validate.py. Verified output:

before: reported share 41%, median first report 38 min
after: reported share 78%, median first report 9 min

38.3 Mixed evidence

Aiko adds qualitative evidence: interviews with ten staff, the burden survey and the complaint log. Staff describe the report button as easy and the feedback as reassuring, though some still dislike simulations. Burden scores improve slightly. Together the evidence supports continuing the programme with adjustments. She follows the advice of Beautement, Sasse and Wonham (2008) to look at compliance budget effects, and finds that the payment hold adds work at month-end, prompting a review of finance staffing that month. Interview notes are anonymised before analysis.

Worked example

Aiko validates the programme with before-after outcomes, caught fraud attempts and interviews, treating results as consistent with success rather than proof.

Synthetic case. Outcomes are synthetic; Beautement et al. (2008) is real.

Errors and limits

Before-after designs are weak. Attack patterns change year to year.

Exercises

Synthesis

Validation tests purpose in real operation with mixed evidence and cautious claims.

Continue studying — chapter 38 of 50

← Chapter 37 (draft) · Course chapter index · Chapter 39 (draft) →

Worked answers

EX112. Whether the system achieves its purpose in real operation.

EX113. Median time fell from 38 to 9 minutes and reporting share rose from 41% to 78%.

EX114. The payment hold adds work for finance at month-end.