Chapter 34 — Comparative project

Lesson map

Lesson 34 published. Map, example, EX100.

flowchart TD
  N0["Learning outcomes"]
  N1["34.1 Collecting data"]
  N2["34.2 Comparing outcomes"]
  N3["Lab — Reporting outcomes at two library networks"]
  N4["34.3 Balanced conclusions"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX100"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["34.1 Collecting data"]
  N2["34.2 Comparing outcomes"]
  N3["Lab — Reporting outcomes at two library networks"]
  N4["34.3 Balanced conclusions"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX100"]

[Learning outcomes]
    |
    v
[34.1 Collecting data]
    |
    v
[34.2 Comparing outcomes]
    |
    v
[Lab — Reporting outcomes at two library networks]
    |
    v
[34.3 Balanced conclusions]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX100]

Learning outcomes

You will (1) carry out a comparison using planned measures; (2) interpret differences with care; (3) present balanced conclusions.

34.1 Collecting data

Nadia collects six months of anonymised data from both networks. Ashcombe had 22 malicious emails reach staff; Dunmore, with 150 staff, had 27. For each, she records the time to first report or notes that it was never reported. She also runs the survey at both networks, receiving 82 responses at Ashcombe and 95 at Dunmore. She keeps data in a protected folder, checks for errors by comparing totals with IT summaries and records every decision about excluded or unclear items. She logs every exclusion in a notebook.

34.2 Comparing outcomes

The lab computes median time to first report and the share of malicious emails reported at each network. Ashcombe shows a shorter median time and a higher reported share. Survey scores show higher confidence and lower fear of blame at Ashcombe. Nadia notes that Dunmore's click rates in simulations are low, which its managers cite as success, but its staff report real attacks slowly. This suggests that low click rates and fast reporting are different things, a point she discusses carefully.

Lab — Reporting outcomes at two library networks

# Reporting outcomes at two synthetic library networks over six months.
ash = [3, 5, 6, 8, 9, 10, 11, 12, 12, 14, 15, 18, 21, 25, 30, 38, 44, 52, 66] + [None] * 3
dun = [9, 15, 22, 30, 41, 47, 55, 70, 88, 120, 150, 180, 240, 300] + [None] * 13
for name, d in (("Ashcombe", ash), ("Dunmore", dun)):
    t = sorted(x for x in d if x is not None)
    print(f"{name}: {len(d)} malicious, reported {len(t)} ({len(t) / len(d):.0%}), median first report {t[len(t) // 2]} min")

Run with python3 lab_networks.py. Verified output:

Ashcombe: 22 malicious, reported 19 (86%), median first report 14 min
Dunmore: 27 malicious, reported 14 (52%), median first report 70 min

34.3 Balanced conclusions

Nadia concludes that Ashcombe's approach is associated with faster reporting and greater staff confidence, but she does not claim it caused these differences. Dunmore's staff may face different attacks, and its programme has strengths, such as regular practice. She suggests that Dunmore might gain by adding fast feedback and a no-blame message, and that Ashcombe might learn from Dunmore's regular refreshers. Balanced conclusions respect both organisations and are more likely to be acted on than claims that one approach is simply right.

Worked example

Nadia compares two networks, finds faster reporting and more confidence at Ashcombe and draws balanced conclusions.

Synthetic case. Data are synthetic.

Errors and limits

Six months is short. Networks face different attacks.

Exercises

• EX100 — Lab medians. What median times to first report did the lab show?
  Show marking criteria

  Rubric: 3 = Ashcombe 14 and Dunmore 70 minutes;; 2 = faster;; 1 = vague;; 0 = none.
• EX101 — Reported share. What shares of malicious emails were reported?
  Show marking criteria

  Rubric: 3 = Ashcombe 86% and Dunmore 52%;; 2 = higher;; 1 = vague;; 0 = none.
• EX102 — Click versus report. What do Dunmore's results suggest?
  Show marking criteria

  Rubric: 3 = low clicks and fast reporting differ;; 2 = different;; 1 = same;; 0 = none.

Synthesis

Comparisons should be interpreted cautiously and concluded with balance.

Continue studying — chapter 34 of 40

← Chapter 33 (draft) · Course chapter index · Chapter 35 (draft) →