Chapter 29 — Security governance, risk and compliance: measurement
Lesson map
Lesson 29 published. Map, example, EX085.
flowchart TD
N0["Learning outcomes"]
N1["29.1 What to measure"]
N2["29.2 Pass rates and thresholds"]
N3["Lab — Control pass rates against thresholds"]
N4["29.3 Not hiding weak areas"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX085"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["29.1 What to measure"]
N2["29.2 Pass rates and thresholds"]
N3["Lab — Control pass rates against thresholds"]
N4["29.3 Not hiding weak areas"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX085"]
[Learning outcomes]
|
v
[29.1 What to measure]
|
v
[29.2 Pass rates and thresholds]
|
v
[Lab — Control pass rates against thresholds]
|
v
[29.3 Not hiding weak areas]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX085]
Learning outcomes
You will (1) measure control performance through test results; (2) set thresholds for key risk indicators; (3) report measures without hiding weak areas.
29.1 What to measure
Nadia learns to separate measures of activity, such as the number of reviews held, from measures of performance, such as the share of control tests passed, and measures of outcome, such as excess access found. NIST CSF 2.0 encourages organisations to monitor and improve their cybersecurity outcomes, and ISO/IEC 27001:2022 requires the organisation to determine what is measured and how results are evaluated. She proposes that Ardwell report mainly performance and outcome measures to the board. Excess access found, for instance, tells the board whether risk is shrinking, while counting reviews held only shows effort.
29.2 Pass rates and thresholds
For one quarter, Ardwell tests five control families. The lab computes pass rates and applies thresholds agreed with management: green at ninety-five percent or more, amber from eighty percent, red below that. Access passes forty-five of forty-eight tests, ninety-four percent, just amber. Patching passes thirty-one of forty, seventy-eight percent, red. Backup passes all twelve. Logging is amber at eighty-seven percent and change management is red at seventy-two percent. Overall, one hundred and thirty-two of one hundred and fifty-five pass. That overall eighty-five percent is the figure the next section warns against reporting on its own.
Lab — Control pass rates against thresholds
"""Control test pass rates and key risk indicator thresholds (synthetic quarter)."""
tests = {"access": (48, 45), "patching": (40, 31), "backup": (12, 12), "logging": (30, 26), "change": (25, 18)}
for fam, (n, ok) in tests.items():
r = ok/n
status = "green" if r >= 0.95 else "amber" if r >= 0.80 else "red"
print(f"{fam}: {ok}/{n} = {r:.0%} {status}")
n = sum(v[0] for v in tests.values()); ok = sum(v[1] for v in tests.values())
print(f"overall {ok}/{n} = {ok/n:.0%}")
Run with python3 lab_kri.py. Verified output:
access: 45/48 = 94% amber
patching: 31/40 = 78% red
backup: 12/12 = 100% green
logging: 26/30 = 87% amber
change: 18/25 = 72% red
overall 132/155 = 85%
29.3 Not hiding weak areas
The overall figure of eighty-five percent looks reasonable, but it hides two red families. Nadia reports each family separately, leading with the reds and their causes: patching delays on substation equipment and emergency changes without review. She warns that thresholds influence behaviour; if teams are judged only on pass rates, they may test easy cases. She recommends rotating test samples and reviewing how tests are chosen, so the measure stays honest over time. Rotating samples prevents patching tests from always landing on office machines and missing substation equipment.
Worked example
Nadia distinguishes activity, performance and outcome measures, computes five pass rates with thresholds and reports red families first.
Synthetic case. Test results and thresholds are synthetic; CSF 2.0 and ISO/IEC 27001:2022 are real.
Errors and limits
Aggregates hide weak areas. Measures invite gaming.
Exercises
- EX085 — Measure kinds. Which three kinds of measure did Nadia separate? Rubric: 3 = activity, performance and outcome;; 2 = three;; 1 = vague;; 0 = none.
- EX086 — Red families. Which two control families were red? Rubric: 3 = patching 78% and change 72%;; 2 = two reds;; 1 = backup;; 0 = none.
- EX087 — Aggregate risk. Why is the overall 85% misleading? Rubric: 3 = it hides two red families;; 2 = hides;; 1 = vague;; 0 = none.
Synthesis
Measures should show performance by family, lead with weaknesses and resist gaming.
Continue studying — chapter 29 of 50
← Chapter 28 (draft) · Course chapter index · Chapter 30 (draft) →