Chapter 9 — Evidence and measurement
Lesson map
Lesson 09 published. Map, example, EX025.
flowchart TD
N0["Learning outcomes"]
N1["9.1 Choosing measures"]
N2["9.2 Computing times"]
N3["Lab — Median time to first report and to verdict"]
N4["9.3 Fair presentation"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX025"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["9.1 Choosing measures"]
N2["9.2 Computing times"]
N3["Lab — Median time to first report and to verdict"]
N4["9.3 Fair presentation"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX025"]
[Learning outcomes]
|
v
[9.1 Choosing measures]
|
v
[9.2 Computing times]
|
v
[Lab — Median time to first report and to verdict]
|
v
[9.3 Fair presentation]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX025]
Learning outcomes
You will (1) choose measures for a reporting service; (2) compute time to first report and verdict time; (3) present measures fairly.
9.1 Choosing measures
Lucas and the practice manager choose measures for Brightwell's reporting: the share of malicious emails reported by at least one person, the time from arrival to first report, the time from report to verdict and staff burden from a short quarterly question. They avoid ranking clinics by click rates in simulations, because rankings embarrass staff and click rates depend heavily on how hard a simulated email was. Measures should help improve the service, not judge individuals. Each measure has a short written definition.
9.2 Computing times
Lucas uses the data model from chapter 4 to compute times for a month of synthetic malicious emails. For each cluster, he finds the earliest report and the verdict time. The lab prints the median time to first report and median time to verdict. He finds that reports usually arrive quickly but verdicts are slow because the IT contractor checks the mailbox only twice a day. This points to a practical fix: notifications for new reports. He shares the result with the contractor.
Lab — Median time to first report and to verdict
# Median minutes: arrival to first report, and report to verdict (synthetic month).
first = [3, 5, 6, 8, 9, 11, 14, 18, 25, 40, 55]
verdict = [60, 95, 120, 150, 170, 190, 210, 260, 300, 340, 420]
m = lambda x: sorted(x)[len(x) // 2]
print(f"emails {len(first)}; median to first report {m(first)} min; median report to verdict {m(verdict)} min")
Run with python3 lab_times.py. Verified output:
emails 11; median to first report 11 min; median report to verdict 190 min
9.3 Fair presentation
Lucas presents measures by groups of clinics with at least twenty staff, not by individual or small clinic. He shows medians with the number of emails behind them. He explains to managers that one month of data is noisy and that trends matter more than single values. Fair presentation protects staff privacy and helps managers draw sensible conclusions instead of reacting to random variation. He also shows the same summary to staff at their monthly meeting, so that everyone sees the same figures as managers and can ask questions about them.
Worked example
Lucas selects reporting measures, computes medians showing slow verdicts and presents grouped, noisy-aware results.
Synthetic case. Data are synthetic.
Errors and limits
Unreported emails are missed. Monthly samples are small.
Exercises
- EX025 — Ranking avoided. Why avoid clinic rankings by click rate? Rubric: 3 = they embarrass staff and depend on difficulty;; 2 = fairness;; 1 = vague;; 0 = none.
- EX026 — Lab medians. What medians did the lab compute? Rubric: 3 = 11 minutes to report and 190 minutes to verdict;; 2 = slow verdict;; 1 = vague;; 0 = none.
- EX027 — Verdict fix. What fix does the slow verdict suggest? Rubric: 3 = notifications for new reports;; 2 = alerts;; 1 = vague;; 0 = none.
Synthesis
Good measures focus on reporting and verdict speed and are presented fairly.
Continue studying — chapter 09 of 30
← Chapter 08 (draft) · Course chapter index · Chapter 10 (draft) →