Chapter 29 — Security architecture and engineering: measurement
Lesson map
Lesson 29 published. Map, example, EX085.
flowchart TD
N0["Learning outcomes"]
N1["29.1 Metrics tied to goals"]
N2["29.2 Computing metrics"]
N3["Lab — Revocation timeliness and coverage"]
N4["29.3 Metrics that mislead"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX085"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["29.1 Metrics tied to goals"]
N2["29.2 Computing metrics"]
N3["Lab — Revocation timeliness and coverage"]
N4["29.3 Metrics that mislead"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX085"]
[Learning outcomes]
|
v
[29.1 Metrics tied to goals]
|
v
[29.2 Computing metrics]
|
v
[Lab — Revocation timeliness and coverage]
|
v
[29.3 Metrics that mislead]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX085]
Learning outcomes
You will (1) define security metrics tied to architecture goals; (2) compute coverage and timeliness metrics from synthetic data; (3) avoid metrics that reward the wrong behaviour.
29.1 Metrics tied to goals
Daniel asks Sofia to propose metrics for the back-office architecture. She starts from goals: authentic tariff tables, timely revocation of retired devices, protected personal data and quick recovery. For each she defines one or two metrics: percentage of tariff loads with verified signatures; median days from device retirement to certificate revocation; percentage of personal-data stores with encryption and access logging enabled; and time to restore ingestion in recovery tests. Each metric has a definition, data source, target and owner. Metrics without goals tend to measure what is easy rather than what matters.
29.2 Computing metrics
Sofia writes a script that reads synthetic records of device retirements and revocations and data-store configurations, and computes the metrics. The lab below shows that revocation timeliness is poor, with a long tail of devices revoked weeks after retirement and some never revoked, while encryption coverage is high. She presents counts and denominators alongside percentages. The numbers immediately prioritise work: the revocation process, not encryption, needs attention. Measurement turns the threat found in chapter 23 into a tracked improvement with a baseline.
Lab — Revocation timeliness and coverage
# Synthetic device retirement/revocation records and data-store configs.
import statistics
delays = [1, 2, 2, 3, 4, 5, 6, 7, 8, 9, 9, 12, 15, 21, 28, 35, 44, None, None, None] # None = never revoked
done = [d for d in delays if d is not None]
print(f"retired devices: {len(delays)}, revoked: {len(done)}, never revoked: {delays.count(None)}")
print(f"median days to revoke (revoked only): {statistics.median(done)}")
print(f"revoked within 7 days: {sum(d <= 7 for d in done)}/{len(delays)}")
stores = {"journeys": (True, True), "refunds": (True, True), "staff": (True, False), "reports": (True, True), "archive": (True, True)}
cov = sum(e and l for e, l in stores.values())
print(f"personal-data stores encrypted+logged: {cov}/{len(stores)}")
Run with python3 lab_metrics.py. Verified output:
retired devices: 20, revoked: 17, never revoked: 3
median days to revoke (revoked only): 8
revoked within 7 days: 8/20
personal-data stores encrypted+logged: 4/5
29.3 Metrics that mislead
Metrics shape behaviour. If the team is rewarded for closing vulnerability tickets quickly, they may close tickets without verification. If the metric is number of controls implemented, they may implement easy controls rather than important ones. Sofia proposes pairing metrics: closure speed with reopen rate; control count with coverage of high-priority risks. She also warns against reporting a metric without its denominator or trend. Anderson (2020) discusses how security metrics can be gamed. Good metrics are few, tied to goals and reviewed to see whether they still drive the intended behaviour.
Worked example
Sofia defines five goal-linked metrics, computes revocation timeliness and encryption coverage from synthetic data and pairs metrics to resist gaming.
Synthetic case. Metrics data are synthetic; Anderson (2020) is real.
Errors and limits
Metrics measure proxies, not security itself. Small denominators make percentages unstable.
Exercises
- EX085 — Goal-linked metric. Give a metric for the goal of timely revocation of retired devices. Rubric: 3 = median days from retirement to revocation;; 2 = metric;; 1 = count of devices;; 0 = none.
- EX086 — Lab revocation result. What did the lab show for revocation timeliness? Rubric: 3 = median 8 days for revoked devices, 3 of 20 never revoked, only 8 of 20 within a week;; 2 = both;; 1 = good;; 0 = none.
- EX087 — Paired metric. Which metric should be paired with ticket closure speed? Rubric: 3 = reopen rate;; 2 = reopen;; 1 = vague;; 0 = none.
Synthesis
Security metrics should follow goals, show denominators and be paired so they reward real improvement.
Continue studying — chapter 29 of 50
← Chapter 28 (draft) · Course chapter index · Chapter 30 (draft) →