Chapter 11 — Supervised laboratory

Lesson map

Lesson 11 published. Map, example, EX051.

flowchart TD
  N0["Learning outcomes"]
  N1["11.1 The test"]
  N2["11.2 Comparing results"]
  N3["Lab — Correct judgements after verdict versions A an"]
  N4["11.3 Adoption"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX051"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["11.1 The test"]
  N2["11.2 Comparing results"]
  N3["Lab — Correct judgements after verdict versions A an"]
  N4["11.3 Adoption"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX051"]

[Learning outcomes]
    |
    v
[11.1 The test]
    |
    v
[11.2 Comparing results]
    |
    v
[Lab — Correct judgements after verdict versions A an]
    |
    v
[11.3 Adoption]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX051]

Learning outcomes

You will (1) test a change to verdict messages with volunteers; (2) compare understanding between versions; (3) decide on adoption.

11.1 The test

In the supervised laboratory, Grace tests two versions of the verdict message. Version A says only whether the message was malicious. Version B adds one sentence explaining the clue, such as a lookalike address or an unusual payment request. Twenty volunteers, who know the purpose, each read verdicts for five synthetic messages in a test environment, half seeing version A and half version B. Afterwards, each judges three new synthetic messages. Alan supervises, and volunteers can withdraw at any time. Each session lasts twenty minutes.

11.2 Comparing results

The lab compares correct judgements on the new messages between the two groups. Volunteers who read version B judge more messages correctly than those who read version A. Grace notes the small sample and that volunteers knew they were being asked to judge messages, which may make them more careful than usual. Still, the result is consistent with the idea that brief explanations help people learn from each report. She records the counts and the difference without overstating them. Alan reviews the analysis.

Lab — Correct judgements after verdict versions A and B

# Correct judgements of three new messages by ten volunteers per version (synthetic).
a = [2, 1, 2, 2, 1, 2, 3, 2, 1, 2]; b = [3, 2, 3, 3, 2, 3, 2, 3, 2, 2]
print(f"version A: {sum(a)}/30 correct; version B: {sum(b)}/30 correct")

Run with python3 lab_versions.py. Verified output:

version A: 18/30 correct; version B: 25/30 correct

11.3 Adoption

Grace recommends adopting version B, since it costs only one sentence per verdict and may help staff learn. She updates templates with common clues and trains her backup to write short explanations. She plans to check after three months whether repeat reporting improves, using real data. Alan notes that she has followed a sound process: a clear question, a small fair test, honest analysis and a cheap change with follow-up, which is how practitioners should improve services. Managers approve the change.

Worked example

Grace tests two verdict versions with twenty volunteers, finds better judgements after explanations and adopts version B with follow-up.

Synthetic case. The test is synthetic.

Errors and limits

Small sample. Volunteers knew the task.

Exercises

• EX051 — Version B. What does version B add?
  Show marking criteria

  Rubric: 3 = one sentence explaining the clue;; 2 = explanation;; 1 = vague;; 0 = none.
• EX052 — Lab result. What did the lab show for correct judgements?
  Show marking criteria

  Rubric: 3 = A 18 of 30, B 25 of 30;; 2 = B better;; 1 = vague;; 0 = none.
• EX053 — Caution. Why might volunteers be more careful than usual?
  Show marking criteria

  Rubric: 3 = they knew they would judge messages;; 2 = awareness;; 1 = vague;; 0 = none.
• EX054 — Cost. What does version B cost?
  Show marking criteria

  Rubric: 3 = one sentence per verdict;; 2 = low;; 1 = vague;; 0 = none.
• EX055 — Follow-up. What follow-up did Grace plan?
  Show marking criteria

  Rubric: 3 = check repeat reporting after three months;; 2 = follow-up;; 1 = vague;; 0 = none.

Synthesis

Small, fair tests support cheap improvements when followed up with real data.

Continue studying — chapter 11 of 12

← Chapter 10 (draft) · Course chapter index · Chapter 12 (draft) →