Chapter 15 — Evidence and measurement
Lesson map
Lesson 15 published. Map, example, EX043.
flowchart TD
N0["Learning outcomes"]
N1["15.1 What to measure"]
N2["15.2 Computing coverage"]
N3["Lab — Control coverage by department"]
N4["15.3 Evidence in context"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX043"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["15.1 What to measure"]
N2["15.2 Computing coverage"]
N3["Lab — Control coverage by department"]
N4["15.3 Evidence in context"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX043"]
[Learning outcomes]
|
v
[15.1 What to measure]
|
v
[15.2 Computing coverage]
|
v
[Lab — Control coverage by department]
|
v
[15.3 Evidence in context]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX043]
Learning outcomes
You will (1) choose measurements that show whether controls work; (2) compute coverage metrics from synthetic data; (3) present evidence with context.
15.1 What to measure
Hannah asks Lucas how Fernway can know whether its controls work. Counting controls says little. He proposes measuring outcomes and coverage: the share of staff accounts with phishing-resistant authentication, the share of laptops meeting the baseline, the time from leaving to account disablement and the number of exposed services found in monthly scans. Each measure links to a risk from earlier chapters. He writes definitions precisely, for example what counts as meeting the baseline, so the numbers mean the same thing each month and cannot be inflated by changing the rules.
15.2 Computing coverage
Lucas writes a script that reads synthetic exports from the identity system and device management tool and computes coverage for each measure, broken down by department. The lab below shows overall figures and the department with the weakest coverage: warehouse staff have the lowest share of phishing-resistant authentication because many share scanners and lack individual devices. A department breakdown turns a single average into an action: the warehouse needs a different authentication design, perhaps badge-based login on shared devices, rather than a general reminder to enrol.
Lab — Control coverage by department
# Coverage of phishing-resistant auth and laptop baseline by department (synthetic export).
staff = {"finance": (12, 11), "buying": (9, 7), "stores": (40, 22), "warehouse": (15, 2), "it": (5, 5)}
laptops = {"finance": (12, 12), "buying": (9, 8), "stores": (14, 11), "warehouse": (3, 2), "it": (6, 6)}
tot = sum(n for n, _ in staff.values()); ok = sum(k for _, k in staff.values())
print(f"phishing-resistant auth: {ok}/{tot} = {ok/tot:.0%}")
for d, (n, k) in sorted(staff.items(), key=lambda x: x[1][1] / x[1][0]): print(f" {d:9} {k}/{n} = {k/n:.0%}")
tl = sum(n for n, _ in laptops.values()); bl = sum(k for _, k in laptops.values())
print(f"laptops meeting baseline: {bl}/{tl} = {bl/tl:.0%}")
Run with python3 lab_coverage.py. Verified output:
phishing-resistant auth: 47/81 = 58%
warehouse 2/15 = 13%
stores 22/40 = 55%
buying 7/9 = 78%
finance 11/12 = 92%
it 5/5 = 100%
laptops meeting baseline: 39/44 = 89%
15.3 Evidence in context
Lucas presents results to Fernway's management with counts, percentages and trends, and he explains what each figure does and does not show. High baseline coverage does not prove laptops are secure, only that they meet the baseline; the baseline may itself miss risks. Leaver disablement times show process performance, not whether former staff misused access. He also lists data quality issues, such as devices missing from the management tool. Measurement builds trust when its limits are stated alongside its results.
Worked example
Lucas defines four coverage and timeliness measures, computes them by department and presents results with limits and data-quality notes.
Synthetic case. Exports and figures are synthetic.
Errors and limits
Coverage metrics measure presence of controls, not their effectiveness. Missing devices inflate apparent coverage.
Exercises
- EX043 — Precise definition. Why define meeting the baseline precisely? Rubric: 3 = so numbers mean the same each month and cannot be inflated by redefinition;; 2 = consistency;; 1 = vague;; 0 = none.
- EX044 — Weakest department. Which department had the weakest phishing-resistant coverage in the lab, and why? Rubric: 3 = warehouse, 2 of 15, shared scanners without individual devices;; 2 = warehouse;; 1 = finance;; 0 = none.
- EX045 — Coverage caution. What does high baseline coverage not prove? Rubric: 3 = that laptops are secure; only that they meet the baseline;; 2 = limit;; 1 = proves security;; 0 = none.
Synthesis
Measurement shows whether controls are in place and working when it is precise, broken down and presented with its limits.
Continue studying — chapter 15 of 40
← Chapter 14 (draft) · Course chapter index · Chapter 16 (draft) →