Chapter 15 — Evidence and measurement
Lesson map
Lesson 15 published. Map, example, EX029.
flowchart TD
N0["Learning outcomes"]
N1["15.1 What makes a measurement useful"]
N2["15.2 Readiness indicators"]
N3["15.3 Provenance"]
N4["15.5 Practical activity"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX029"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["15.1 What makes a measurement useful"]
N2["15.2 Readiness indicators"]
N3["15.3 Provenance"]
N4["15.5 Practical activity"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX029"]
[Learning outcomes]
|
v
[15.1 What makes a measurement useful]
|
v
[15.2 Readiness indicators]
|
v
[15.3 Provenance]
|
v
[15.5 Practical activity]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX029]
Learning outcomes
You will (1) define validity, reliability and coverage for security measurements; (2) choose indicators for cryptographic readiness; (3) record provenance for every observation; and (4) present measurements with their blind spots.
15.1 What makes a measurement useful
A measurement is valid if it measures what it claims, reliable if it gives the same result when repeated, and has known coverage if we know what share of the population it observes. "Percentage of TLS connections using hybrid key exchange" is valid if counted from ServerHello messages, reliable if the sensor is stable, and its coverage is the share of traffic passing the sensor. Without coverage, a 90 % figure may describe only the easy tenth of the estate.
Coverage is the property most often missing from security dashboards.
15.2 Readiness indicators
Useful indicators for a quantum migration include: share of assets with a known cryptographic inventory; share of long-lived data flows protected by hybrid key exchange; share of certificates issued by CAs that can issue post-quantum certificates; number of vendors with a published PQC roadmap; and mean time to change an algorithm by configuration. Each indicator should have an owner, a source and a refresh frequency.
HCU began with three of these and added the others once owners were agreed. Each has a source.
15.3 Provenance
Record for every observation: what was observed, by which method (passive network, configuration, code scan, vendor statement), when, and by whom. Methods differ in confidence: an observed handshake is stronger evidence than a vendor's datasheet. In TLS 1.3 a passive sensor sees the selected group but not the server certificate, which is encrypted, so certificate algorithms must come from endpoint or configuration evidence. Mixing methods without labels makes dashboards look more certain than they are.
HCU's inventory has a column for method and one for date, and dashboards show both.
15.4 Worked example
HCU's first dashboard reports "92 % of connections hybrid". On inspection, the only sensor sits in front of the new web gateway, which already uses X25519MLKEM768; branch VPNs and the ATM network are invisible to it. The corrected dashboard shows "web traffic: 92 % hybrid (sensor coverage 100 % of web gateway); VPN: 0 % (configuration evidence, 40/40 branches); ATM: unknown (no sensor)". The headline number fell, and the decision quality rose.
The correction did not change any system, only what the dashboard claimed. Managers could now see where work remained and where the team simply lacked sight. The ATM row led to a request to the ATM vendor for configuration evidence, since adding a network sensor there was not possible.
15.5 Practical activity
Design three readiness indicators for HCU. For each, give definition, source, coverage, owner and refresh rate, and name one way it could mislead.
A good definition says exactly what is counted and what is not. Coverage should state which part of the estate the source can see. An owner is a person or role who will notice when the figure looks wrong. The way an indicator could mislead is often its most useful line: for example, a count of vendors with roadmaps says nothing about whether the roadmaps are realistic or have dates.
15.6 Common errors and limits
- Reporting a percentage without its denominator and coverage.
- Treating vendor claims as observations.
- Counting connections when the decision concerns assets or data classes.
A percentage without its base can hide a tiny sample or a biased one. Vendor statements are useful but untested; label them as claims until someone observes the behaviour. Counting connections favours busy systems, while migration decisions usually concern which assets or data classes are protected. A single busy web service can make the figure look good while a quiet but sensitive archive link stays classical. Choose the unit that matches the decision.
Exercises
- EX029 — Coverage. Rewrite "95 % of our certificates are ready for PQC" so that it states method, coverage and denominator. Rubric: 3 = all three present and plausible.
- EX030 — Indicator design. Propose an indicator for "ability to change algorithms quickly" and explain how to measure it. Rubric: 3 = definition, measurement method, limitation.
Synthesis
Valid, reliable measurements with stated coverage and provenance let decision-makers see both progress and blind spots.
Continue studying — chapter 15 of 40
← Chapter 14 (draft) · Course chapter index · Chapter 16 (draft) →