Chapter 9 — Evidence and measurement
Lesson map
Lesson 09 published. Map, example, EX025.
flowchart TD
N0["Learning outcomes"]
N1["9.1 Choosing measures"]
N2["Lab — Monthly security health figures from exports"]
N3["9.2 Computing from exports"]
N4["9.3 Reporting trends"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX025"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["9.1 Choosing measures"]
N2["Lab — Monthly security health figures from exports"]
N3["9.2 Computing from exports"]
N4["9.3 Reporting trends"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX025"]
[Learning outcomes]
|
v
[9.1 Choosing measures]
|
v
[Lab — Monthly security health figures from exports]
|
v
[9.2 Computing from exports]
|
v
[9.3 Reporting trends]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX025]
Learning outcomes
You will (1) choose simple measures of security health; (2) compute them from exports; (3) report trends to managers.
9.1 Choosing measures
Sam asks Tomás how Brightwater can tell if it is improving. Tomás proposes five measures: share of devices updated within thirty days, share of staff using individual sign-in, number of inbound internet rules, days from leaving to account disablement and number of unowned devices. Each is cheap to collect from existing exports. He writes a one-line definition for each so that the figure means the same thing every month, and he records where the data comes from and who produces it.
Lab — Monthly security health figures from exports
# Monthly security health figures from synthetic exports.
import datetime as dt
today = dt.date(2026, 3, 31)
updated = [dt.date(2026, 3, d) for d in range(3, 32)] + [dt.date(2026, 1, 15)] * 31 # 60 devices
owners = ["sam"] * 57 + [None] * 3
leavers = [(dt.date(2026, 3, 2), dt.date(2026, 3, 3)), (dt.date(2026, 3, 9), dt.date(2026, 3, 16)),
(dt.date(2026, 3, 20), dt.date(2026, 3, 24))]
recent = sum((today - d).days <= 30 for d in updated)
print(f"updated within 30 days: {recent}/{len(updated)} = {recent/len(updated):.0%}")
print(f"unowned devices: {owners.count(None)}")
days = [(off - left).days for left, off in leavers]
print(f"days to disable leavers: {days}, average {sum(days)/len(days):.0f}")
Run with python3 lab_health.py. Verified output:
updated within 30 days: 29/60 = 48%
unowned devices: 3
days to disable leavers: [1, 7, 4], average 4
9.2 Computing from exports
Tomás writes a script that reads synthetic device and account exports and computes the measures. The lab shows the current month: twenty-nine of sixty devices updated within thirty days, three unowned devices and an average of four days to disable leavers. He checks a sample of records by hand to make sure the script reads dates correctly. A small error in date parsing would make every month look better or worse than reality, undermining trust in the whole report. He also keeps the raw exports for a month so figures can be rechecked.
9.3 Reporting trends
Tomás presents three months of figures to the operations manager as a simple table with arrows showing direction. He explains what each figure means and what action it suggests. The update share is the weakest figure, which supports the monthly update window agreed in chapter 6. He avoids colour-coded scores that hide detail. The manager asks for the report monthly. Trends matter more than single numbers, because they show whether changes are working and give early warning when something slips.
Worked example
Tomás defines five health measures, computes them from synthetic exports with a hand-checked sample and reports a three-month trend.
Synthetic case. Exports and figures are synthetic.
Errors and limits
Measures can be gamed once targets are set. Exports may miss devices.
Exercises
- EX025 — Measure definitions. Why write a one-line definition for each measure? Rubric: 3 = so it means the same thing every month;; 2 = consistency;; 1 = vague;; 0 = none.
- EX026 — Update share. What was the update share in the lab? Rubric: 3 = 29 of 60, 48%;; 2 = 48%;; 1 = vague;; 0 = none.
- EX027 — Hand checking. Why check a sample of records by hand? Rubric: 3 = to confirm dates are read correctly;; 2 = parsing;; 1 = vague;; 0 = none.
Synthesis
A few well-defined measures computed from exports and shown as trends tell managers whether security is improving.
Continue studying — chapter 09 of 30
← Chapter 08 (draft) · Course chapter index · Chapter 10 (draft) →