Chapter 15 — Evidence and measurement
Lesson map
Lesson 15 published. Map, example, EX043.
flowchart TD
N0["Learning outcomes"]
N1["15.1 What to measure"]
N2["15.2 Reliable evidence"]
N3["15.3 Before and after"]
N4["15.4 Gaming indicators"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX043"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["15.1 What to measure"]
N2["15.2 Reliable evidence"]
N3["15.3 Before and after"]
N4["15.4 Gaming indicators"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX043"]
[Learning outcomes]
|
v
[15.1 What to measure]
|
v
[15.2 Reliable evidence]
|
v
[15.3 Before and after]
|
v
[15.4 Gaming indicators]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX043]
Learning outcomes
You will (1) define measurable security indicators; (2) collect evidence reliably; (3) interpret change.
15.1 What to measure
Security is hard to measure directly, so practitioners use indicators that relate to it. Priya chooses indicators for the clinics: the number of group pairs that can reach each other, the share of devices with a known owner, the share of servers using current TLS versions and the number of denied guest-to-internal connections per week. Each indicator links to a risk from her table. Indicators should be cheap to collect, understandable to managers and hard to improve without actually improving security.
15.2 Reliable evidence
Evidence is reliable if someone else could collect it again and get the same result. Priya documents each measurement: the data source, the date range, the script used and its version. She keeps raw data for the project's duration, so figures can be rechecked. Screenshots and verbal reports are weak evidence because they cannot be repeated. This discipline matters most when results are surprising, since a surprising result that cannot be reproduced is likely to be challenged, and rightly so.
15.3 Before and after
To show that a change helped, Priya measures before and after, under similar conditions. If denied guest connections fall after separating networks, she checks that guest usage did not simply fall at the same time, perhaps because of a holiday. Comparing rates rather than counts helps, as does measuring for several weeks either side of the change. Without this care, normal variation can be mistaken for an effect, and money can be spent repeating a change that did nothing. She also notes any other changes made during the same period.
15.4 Gaming indicators
Any indicator can be improved in ways that do not help security. If the indicator is "number of open alerts", closing alerts without investigating them improves it. Priya pairs indicators so that gaming one shows in another: open alerts alongside a sample review of closed alerts. She also explains each indicator's purpose to the people measured by it. Indicators work best as information for improvement, not as targets with rewards attached, which tends to encourage gaming. She reviews the indicator set each term and retires indicators that have stopped being useful.
Worked example
Priya's indicator sheet lists four indicators, each with source, script, frequency and linked risk.
Synthetic case. Indicators are synthetic.
Errors and limits
Comparing a single week before and after a change confuses normal variation with effect.
Exercises
- EX043 — Indicator qualities. Name three qualities of a good security indicator. Rubric: 3 = cheap, understandable, hard to game or linked to risk;; 2 = two;; 1 = one;; 0 = none.
- EX044 — Holiday effect. Why check guest usage when denied guest connections fall? Rubric: 3 = fall may reflect fewer guests, not the control;; 2 = other causes;; 1 = vague;; 0 = none.
- EX045 — Reproducibility record. What should be recorded for each measurement? Rubric: 3 = source, date range, script and version;; 2 = source;; 1 = result only;; 0 = none.
Synthesis
Well-chosen, reproducible indicators provide evidence that security actually improved.
Continue studying — chapter 15 of 40
← Chapter 14 (draft) · Course chapter index · Chapter 16 (draft) →