Chapter 24 — Human factors and cyber policy: methods I

Chapter 24 — Human factors and cyber policy: methods I

Lesson map

Lesson 24 published. Map, example, EX070.

flowchart TD
  N0["Learning outcomes"]
  N1["24.1 Cognitive walkthrough"]
  N2["24.2 A responsible survey"]
  N3["Lab — Barrier counts and shares from survey response"]
  N4["24.3 Combining methods"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX070"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["24.1 Cognitive walkthrough"]
  N2["24.2 A responsible survey"]
  N3["Lab — Barrier counts and shares from survey response"]
  N4["24.3 Combining methods"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX070"]

[Learning outcomes]
    |
    v
[24.1 Cognitive walkthrough]
    |
    v
[24.2 A responsible survey]
    |
    v
[Lab — Barrier counts and shares from survey response]
    |
    v
[24.3 Combining methods]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX070]

Learning outcomes

You will (1) run a cognitive walkthrough of a security task; (2) conduct a short survey responsibly; (3) combine methods to understand a problem.

24.1 Cognitive walkthrough

Aiko uses a cognitive walkthrough on Greyford's password reset process. She steps through each action a user must take, asking whether the user will know what to do, see how to do it, understand feedback and know they have succeeded. She finds that the reset page asks for an employee number most field staff do not remember, and the confirmation message is ambiguous. Walkthroughs are quick, need no participants and find many problems early, though they rely on the evaluator imagining the user accurately.

24.2 A responsible survey

Aiko surveys staff about reporting barriers. The survey is anonymous, short and voluntary, and it asks about experiences rather than testing knowledge. She avoids leading questions and includes an open question for comments. She reports results only for groups of twenty or more. One hundred and eighty staff respond. The most common barrier is not knowing whether something is suspicious enough to report, followed by not knowing how to report on a phone. She shares the headline results with all staff within two weeks, thanking them and explaining what will change, so that respondents see their answers had an effect.

Lab — Barrier counts and shares from survey responses

# Count reporting barriers from an anonymous survey (synthetic multi-select, n=180).
n = 180
barriers = {"unsure if suspicious enough": 71, "don't know how to report on phone": 48, "worried about looking foolish": 33,
            "no time": 29, "never get feedback": 41, "didn't know reporting existed": 12}
for b, c in sorted(barriers.items(), key=lambda x: -x[1]): print(f"{b:34} {c:3d}/{n} = {c/n:.0%}")

Run with python3 lab_survey.py. Verified output:

unsure if suspicious enough         71/180 = 39%
don't know how to report on phone   48/180 = 27%
never get feedback                  41/180 = 23%
worried about looking foolish       33/180 = 18%
no time                             29/180 = 16%
didn't know reporting existed       12/180 = 7%

24.3 Combining methods

The walkthrough found usability problems; the survey found uncertainty about when to report. Together they suggest two fixes: improve the reset and reporting interfaces, and tell staff clearly that any doubt is a good reason to report. Aiko notes that neither method alone would have shown both. She plans usability testing with real users next, since walkthroughs and surveys describe problems but do not show how people actually behave when using the system. She records how each finding was obtained, so readers know which claims rest on expert review and which on staff reports, and she schedules the usability tests for the following month.

Worked example

Aiko walks through a reset task finding two problems, surveys reporting barriers responsibly and combines both methods into two fixes.

Synthetic case. Survey and findings are synthetic.

Errors and limits

Walkthroughs depend on evaluator imagination. Surveys measure perceptions, not behaviour.

Exercises

• EX070 — Walkthrough question. What four questions guide each walkthrough step? Rubric: 3 = know what to do, see how, understand feedback, know success;; 2 = three;; 1 = vague;; 0 = none.
• EX071 — Top barrier. What was the most common reporting barrier in the lab? Rubric: 3 = unsure whether suspicious enough: 71 of 180;; 2 = uncertainty;; 1 = phone;; 0 = none.
• EX072 — Clear message. What message addresses that barrier? Rubric: 3 = any doubt is a good reason to report;; 2 = doubt;; 1 = vague;; 0 = none.

Synthesis

Combining quick evaluation methods with responsible surveys reveals both interface and understanding problems.

Continue studying — chapter 24 of 50

← Chapter 23 (draft) · Course chapter index · Chapter 25 (draft) →