Chapter 15 — Evidence and measurement

Lesson map

Lesson 15 published. Map, example, EX029.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["15.1 What counts as evidence"]
  N2["15.2 Measurement and metrics"]
  N3["15.3 Uncertainty and sensitivity"]
  N4["15.5 Practical activity"]
  N5["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX029"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["15.1 What counts as evidence"]
  N2["15.2 Measurement and metrics"]
  N3["15.3 Uncertainty and sensitivity"]
  N4["15.5 Practical activity"]
  N5["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX029"]

[Learning outcomes and connection]
    |
    v
[15.1 What counts as evidence]
    |
    v
[15.2 Measurement and metrics]
    |
    v
[15.3 Uncertainty and sensitivity]
    |
    v
[15.5 Practical activity]
    |
    v
[Chapter synthesis]
    |
    v
[Worked example] --> [Exercise EX029]

Learning outcomes and connection

After this chapter you can (1) distinguish a measurement, an indicator and a claim; (2) judge evidence by its source, method, recency and independence; (3) design a security metric that is valid, reliable and hard to game; and (4) use sensitivity analysis to decide whether uncertain evidence is good enough for a decision. It builds on Chapters 1, 4 and 12 and prepares Chapter 18 (research methods), Chapter 27 (verification) and Chapter 37 (evidence review).

15.1 What counts as evidence

A claim is a statement that something is true ("our backups can restore the dispatch system within four hours"). Evidence is what supports or undermines it. Evidence varies in strength. A demonstration (a timed restore drill on the production configuration, last month) is stronger than a document (a backup policy) which is stronger than an assertion ("the vendor says it works"). Ask four questions of any evidence: Source — who produced it, and do they have an interest in the answer? Method — how was it produced, and could someone repeat it? Recency — does it still describe the system as it is now? Independence — was it checked by someone other than the person who made the claim?

15.2 Measurement and metrics

A measurement is a value obtained by a defined procedure (restore time in minutes, measured from start to the first successful dispatch transaction). An indicator or metric is a measurement used to track something we care about. Good metrics are valid (they measure what matters — percentage of critical assets with a tested restore, not number of backup jobs), reliable (two people measuring get the same result), timely and hard to game. Goodhart's warning applies: once a measure becomes a target, people optimise the measure. A patch metric of "percentage of systems patched within 30 days" encourages excluding hard systems from the denominator; report the denominator and the exclusions.

15.3 Uncertainty and sensitivity

Most security evidence is uncertain: small samples (Chapter 12), estimates (Chapter 1), proxies. Sensitivity analysis asks how much the decision changes when an uncertain input changes. If a recommendation holds across the plausible range, the evidence is adequate for that decision even if imprecise. If it flips, either gather better evidence on that input or make the decision robust (choose the option that is acceptable across the range). This is a practical antidote to both false precision and paralysis.

Lab 1 is the model: B stays ahead when any one probability is halved or doubled.

15.4 Worked example: is CEDC's dispatch restorable? (synthetic)

This case is synthetic. CEDC's board asks: "Can we restore computer-aided dispatch within four hours after ransomware?" Available evidence:

Lesson table; scroll horizontally to read all columns →

Assessment: no evidence demonstrates the claim — a restore of the whole dispatch service, including application servers, integrations and credentials, under ransomware conditions (where the backup server itself may be encrypted). The honest answer to the board is "unknown, with partial evidence that the database alone restores within four hours". Recommendation: an end-to-end restore drill into an isolated environment, observed by internal audit, measuring time to first successful test dispatch; and confirmation that at least one backup copy is offline or immutable.

15.5 Practical activity

Take one claim about a service you use ("my phone backs up my photos"). List the evidence you have, classify each item by type, source, recency and independence, and design one test that would demonstrate the claim. Do not run anything on systems you do not own.

For a phone backup, the strongest test is a restore of one photo to another device you own, timed, with the date recorded. A backup app's "last backup: today" is a measurement of the backup, not of the restore.

15.6 Common errors and limits

• Treating backup success as restore evidence.
• Metrics that count activity instead of outcomes.
• Denominators that silently exclude the hardest cases.
• Ignoring the interest of the evidence's source.
• Demanding precision a decision does not need, or accepting confidence it has not earned.

CEDC's evidence table shows the first: 362 of 365 successful jobs say nothing about restoring the application. The second appears when a team reports number of backup jobs instead of tested restores. The third is the patch metric with hard systems left out. The fourth is the vendor's undated "under 3 hours". The fifth cuts both ways: the board does not need minutes, but it cannot accept "probably fine".

Exercises — chapter 15

EX029 — Grade the evidence. Objective: apply the four questions. Inputs: claim "Calder Mutual's card service fails over to its second data centre within 15 minutes"; evidence: architecture diagram (2025), last year's failover test report by the operations team (14 min), auditor's note that the test was announced a week ahead. Steps: grade each item and state what the evidence does and does not show. Expected evidence: table and two sentences. Rubric: 3 = recognises the test as a real but announced, non-independent demonstration and asks for an unannounced or independently observed test under realistic load; 2 = partial; 1 = accepts the claim; 0 = none.

EX030 — Fix a metric. Objective: design a valid, hard-to-game metric. Inputs: "number of phishing-awareness courses completed". Steps: propose a better metric for the risk of credential phishing, define its measurement and name one way it could be gamed. Expected evidence: definition and gaming risk. Rubric: 3 = outcome-oriented metric (e.g. share of privileged accounts with phishing-resistant authentication, or report rate in authorised simulations) with defined procedure and honest gaming risk; 2 = better but loosely defined; 1 = activity metric; 0 = none.

Chapter synthesis

Evidence has a source, a method, a date and a degree of independence. Measure outcomes rather than activity, report denominators, and use sensitivity analysis to decide whether uncertain evidence is good enough. "Unknown, with partial evidence" is often the most honest and useful answer.