OpenCyber Academy
Technologist · Chapter 23 · English
Course code: OCA-075Chapter 23: OCA-075-C023

Chapter 23 — Human factors and cyber policy: evaluation

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 23 published. Map, example, EX067.

flowchart TD
  N0["Learning outcomes"]
  N1["23.1 Goals and results"]
  N2["Lab — Before and after evaluation summary"]
  N3["23.2 Staff views"]
  N4["23.3 Next steps"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX067"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["23.1 Goals and results"]
  N2["Lab — Before and after evaluation summary"]
  N3["23.2 Staff views"]
  N4["23.3 Next steps"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX067"]

[Learning outcomes]
    |
    v
[23.1 Goals and results]
    |
    v
[Lab — Before and after evaluation summary]
    |
    v
[23.2 Staff views]
    |
    v
[23.3 Next steps]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX067]

Learning outcomes

You will (1) evaluate the programme against its goals after six months; (2) combine measures with staff views; (3) recommend next steps.

23.1 Goals and results

Lucas evaluates the programme after six months against its goals: more malicious emails reported, faster reporting and verdicts, payment fraud stopped and low staff burden. The share of malicious emails reported rose from about half to around four fifths. Median verdict time fell from over three hours to about an hour. Two fake bank detail changes were held and refused. Burden scores stayed level. The lab summarises the before and after values side by side for the practice manager's report. Managers receive the summary first.

Lab — Before and after evaluation summary

# Before and after evaluation summary (synthetic).
rows = [("reported share", "48%", "81%"), ("median verdict (min)", "190", "62"), ("fake bank changes held", "0 of 0", "2 of 2"), ("burden (1-5)", "2.9", "2.9")]
for n, b, a in rows: print(f"{n}: before {b}, after {a}")

Run with python3 lab_eval.py. Verified output:

reported share: before 48%, after 81%
median verdict (min): before 190, after 62
fake bank changes held: before 0 of 0, after 2 of 2
burden (1-5): before 2.9, after 2.9

23.2 Staff views

Lucas repeats a short version of the survey and runs two small group discussions. Staff say reporting now feels easy and welcome, and they like hearing back. Receptionists say the callback card helps them refuse pushy callers. Some clinicians still find security keys awkward when moving between rooms. Staff views explain the numbers and highlight problems that measures miss, such as the key issue, which Lucas adds to his recommendations for the next phase of the programme. Group notes contain no names.

23.3 Next steps

Lucas recommends continuing the programme, adding a second key per clinician so they can leave one in each room, and running a short refresher every two months using recent examples. He suggests that Brightwell consider a small number of announced simulations in the next year, under rules agreed with staff, if managers want to check readiness for new attack styles. He presents limits honestly: six months is short, and some improvements may reflect novelty that could fade. The practice manager accepts all of them.

Worked example

Lucas evaluates six months of results, combines them with staff views and recommends extra keys, refreshers and cautious, announced simulations.

Synthetic case. Data are synthetic.

Errors and limits

Novelty effects. Survey samples differ over time.

Exercises

Synthesis

Evaluation combines measures and voices and leads to honest next steps.

Continue studying — chapter 23 of 30

← Chapter 22 (draft) · Course chapter index · Chapter 24 (draft) →

Worked answers

EX067. From 48% to 81%.

EX068. Using security keys when moving between treatment rooms.

EX069. Only announced, under rules agreed with staff.