Chapter 29 — Security operations and blue team: measurement
Lesson map
Lesson 29 published. Map, example, EX085.
flowchart TD
N0["Learning outcomes"]
N1["29.1 Why measure"]
N2["29.2 Choosing metrics"]
N3["Lab — response-time summaries"]
N4["29.3 Reading the result"]
N5["29.4 Distorting metrics"]
N6["29.5 Context"]
N7["29.6 Data quality"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX085"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["29.1 Why measure"]
N2["29.2 Choosing metrics"]
N3["Lab — response-time summaries"]
N4["29.3 Reading the result"]
N5["29.4 Distorting metrics"]
N6["29.5 Context"]
N7["29.6 Data quality"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX085"]
[Learning outcomes]
|
v
[29.1 Why measure]
|
v
[29.2 Choosing metrics]
|
v
[Lab — response-time summaries]
|
v
[29.3 Reading the result]
|
v
[29.4 Distorting metrics]
|
v
[29.5 Context]
|
v
[29.6 Data quality]
|
v
[Worked example] --> [Exercise EX085]
Learning outcomes
You will (1) define SOC metrics linked to goals; (2) compute robust summaries of response times; (3) avoid metrics that distort behaviour; (4) present measurements with context.
29.1 Why measure
Knerler, Parker and Zimmerman (2022) treat measurement as a strategy: "measure performance to improve performance". Metrics show whether changes help and where effort should go.
Without measurement, a SOC cannot tell whether a new rule reduced harm or just added noise, or whether a staffing change helped. Measures also support honest conversations with management about what the team can deliver. The aim is improvement, not judging individuals, which affects how metrics are chosen and shared. Lindqvist starts with a handful of measures and adds more only when a clear question needs answering.
29.2 Choosing metrics
Good SOC metrics tie to goals: time from detection to acknowledgement, share of incidents detected internally rather than reported by customers, coverage of priority techniques, and false-alert rates per rule.
Each metric should answer a question someone actually asks. Time to acknowledgement tells the CISO how quickly the SOC reacts; internal versus customer-reported detection shows whether monitoring finds problems first; technique coverage shows breadth; false-alert rates per rule show where tuning is needed. Collecting metrics nobody uses wastes effort and clutters reports.
Lab — response-time summaries
"""Median and 90th percentile of detection-to-acknowledgement minutes (synthetic)."""
import statistics
minutes = [3, 4, 4, 5, 5, 6, 6, 7, 8, 9, 11, 12, 14, 18, 41]
q = statistics.quantiles(minutes, n=10, method="inclusive")
print(f"n={len(minutes)} mean={statistics.mean(minutes):.1f} median={statistics.median(minutes)} p90={q[8]:.1f}")
Run with python3 lab_latency.py. Verified output:
n=15 mean=10.2 median=7 p90=16.4
29.3 Reading the result
Fifteen acknowledgement times give a mean of 10.2 minutes but a median of 7 and a 90th percentile of 16.4. One 41-minute case pulls the mean up. The median describes the typical case, the 90th percentile the slow tail that matters for the 15-minute requirement.
Reporting only the mean would suggest the team typically takes ten minutes, when most acknowledgements happen in about seven. Reporting only the median would hide that a tenth of cases take over sixteen minutes, beyond the requirement. Amara looks at the 41-minute case separately and finds it arrived during a shift handover, which points to a process fix.
29.4 Distorting metrics
Kokulu et al. (2019) found metric mismatches common in SOCs. Rewarding fast closure can produce careless closure. Pair speed metrics with quality checks, such as reviewed samples of closed alerts.
Once a number becomes a target, people find ways to move it that do not reflect real improvement. If analysts are judged by alerts closed per hour, quick closures without investigation become rational. Using several metrics together makes any single one harder to game. The sample of closed alerts is drawn at random each week, so analysts cannot predict which closures will be reviewed.
29.5 Context
A metric without context misleads: acknowledgement time rose last month because night staffing fell, not because analysts slowed. Reports include such explanations.
A chart showing slower acknowledgements could lead managers to blame analysts or demand overtime. With the explanation that night staffing fell from two to one, the conversation turns to staffing decisions instead. Amara adds a short notes column to the monthly report for events such as staffing changes, incidents and new rules. Reports compare each month with the same month last year where seasonal patterns matter.
29.6 Data quality
Metrics depend on case-system timestamps; analysts who acknowledge late in the tool but act early make data wrong. Training and tooling fix this.
If analysts start investigating from an email notification and only click acknowledge later, the case system records a slower time than reality. The metric then misleads in the opposite direction. Making acknowledgement automatic when an analyst opens the alert, and training staff on why timestamps matter, aligns the data with what actually happened. Amara checks a sample of cases against chat logs.
Coverage metrics
Coverage of priority ATT&CK techniques is measured as the share with at least one tested detection. Untested detections do not count, because a rule that has never fired on a safe test event may not work. Lindqvist reports this share quarterly, with the list of gaps attached so the figure cannot hide which techniques are missing.
Worked example
Amara reports: "Median acknowledgement 7 minutes, 90th percentile 16.4 against a 15-minute target. The slowest case (41 minutes) occurred during a shift handover; we propose overlapping handovers by 15 minutes."
Synthetic case. Times come from the lab script.
Errors and limits
Fifteen values are few; percentiles from small samples are unstable.
Exercises
- EX085 — Lab summaries. Report the mean, median and 90th percentile. Rubric: 3 = 10.2, 7, 16.4; 2 = two; 1 = one; 0 = none.
- EX086 — Why median. Why describe acknowledgement with median and percentile rather than mean? Rubric: 3 = skewed; outlier pulls mean; tail matters for target; 2 = partial; 1 = mean fine; 0 = none.
- EX087 — Distortion guard. How can a speed metric be kept from causing careless closure? Rubric: 3 = pair with quality checks of reviewed closed alerts; 2 = partial; 1 = remove metrics; 0 = none.
Synthesis
Measure what links to goals, summarise robustly, guard against distortion and always report context.
Continue studying — chapter 29 of 50
← Chapter 28 (draft) · Course chapter index · Chapter 30 (draft) →