Chapter 15 — SOC metrics
Lesson map
Lesson 15 published. Map, example, EX057.
flowchart TD
N0["Learning outcomes"]
N1["15.1 Why measure"]
N2["Lab — response times"]
N3["15.2 Lab result"]
N4["15.3 Medians"]
N5["15.4 Bad metrics"]
N6["15.5 Better metrics"]
N7["15.6 Presenting"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX057"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["15.1 Why measure"]
N2["Lab — response times"]
N3["15.2 Lab result"]
N4["15.3 Medians"]
N5["15.4 Bad metrics"]
N6["15.5 Better metrics"]
N7["15.6 Presenting"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX057"]
[Learning outcomes]
|
v
[15.1 Why measure]
|
v
[Lab — response times]
|
v
[15.2 Lab result]
|
v
[15.3 Medians]
|
v
[15.4 Bad metrics]
|
v
[15.5 Better metrics]
|
v
[15.6 Presenting]
|
v
[Worked example] --> [Exercise EX057]
Learning outcomes
You will (1) choose meaningful SOC metrics; (2) compute median times; (3) avoid metrics that drive bad behaviour; (4) present metrics to managers.
15.1 Why measure
Metrics show whether the SOC is improving and where to invest. For Northgate, measurement shows whether changes such as the shift overlap or the playbook change actually helped. Without numbers, the team relies on impressions, which tend to remember dramatic cases and forget routine ones. The lab below measures acknowledge and contain times for six incidents to show what a small set can and cannot say. The same metrics are reported to the council's risk committee each quarter on one page.
Lab — response times
"""Median minutes to acknowledge and to contain for synthetic incidents."""
from statistics import median
inc = [("I1", 4, 35), ("I2", 12, 60), ("I3", 3, 22), ("I4", 25, 140), ("I5", 6, 41), ("I6", 8, 50)]
ack = [a for _, a, _ in inc]; con = [c for _, _, c in inc]
print("median acknowledge:", median(ack), "min")
print("median contain:", median(con), "min")
print("acknowledged within 10 min:", sum(a <= 10 for a in ack), "of", len(ack))
Run with python3 lab_mttr.py. Verified output:
median acknowledge: 7.0 min
median contain: 45.5 min
acknowledged within 10 min: 4 of 6
15.2 Lab result
For six incidents, the median time to acknowledge is 7 minutes and the median time to contain is 45.5 minutes; four of six were acknowledged within 10 minutes. I4, acknowledged after 25 minutes and contained after 140, happened during a shift change. The outlier is why the medians are reported rather than averages. I4 is still reported as its own line, because it shows a process problem worth fixing. Carlos checks whether other slow cases also fell at shift change before recommending the overlap. With six cases, he treats the pattern as a lead rather than proof.
15.3 Medians
Medians resist one very long case dragging the average. In the lab, I4's 140 minutes to contain would pull the mean up to 58 minutes, while the median stays at 45.5. The median describes a typical incident better when one case is unusual. The outlier is still reported separately, since it shows the shift-change problem. With only six incidents, one extreme case would move the mean noticeably. Carlos reports both the median and the outlier, so readers see the typical case and the exception.
15.4 Bad metrics
Counting closed alerts per analyst encourages closing quickly without care. Northgate avoids it. An analyst rewarded for closures might close the parks01 alerts without checking the tablet or linking the alerts. The measure would look good while the work got worse. Northgate prefers measures that reward care, such as the share of alerts with clear notes. Measures are chosen in the monthly meeting, so analysts can say when one creates the wrong incentive. Aisha drops any measure that the team finds is being gamed.
15.5 Better metrics
Time to contain, share of alerts with clear notes, rule precision and percentage of assets sending logs. Each of these measures links to something the SOC itself can change. Time to contain is the measure the playbook change aims at; note quality improves triage handovers; rule precision rose from 1 in 110 to 1 in 7 after tuning; and log coverage tracks the 18 servers found not sending logs. Each one is measured the same way every month, so the trends shown to managers are fair.
15.6 Presenting
One page, trends over months, with explanation of outliers. Trends over months show whether changes worked, which one month cannot. Explaining outliers, such as I4 at shift change, stops readers drawing wrong conclusions from one bad case. Keeping it to one page means the risk committee actually reads it. Carlos writes the page and Aisha checks it before it goes to the committee. The page shows six months of trend data where available, and states clearly when a sample, such as six incidents, is too small.
Data quality for metrics
Metrics are only as good as the times recorded. Carlos finds two incidents where the acknowledge time was entered by hand an hour after the event. Northgate changes the case tool so acknowledge time is captured automatically when an analyst opens the alert, and containment time is captured when the containment action is logged. He also notes that metrics should be compared over the same kinds of cases, since phishing cases are naturally faster to contain than server compromises.
Using metrics for learning
Metrics are discussed in a monthly team meeting, not used to rank people. The team looks at the slowest cases and asks what made them slow: missing information, unclear ownership, or waiting for approvals. In one meeting, the team discovers that containment waited 30 minutes for a manager's approval to isolate an ordinary PC three times; the playbook is changed so analysts can isolate PCs alone, while disabling accounts was already theirs and servers still need the IT manager.
Worked example
Carlos's report notes the I4 outlier came at shift change and recommends an overlap of 15 minutes between shifts. Aisha adopts it.
Synthetic case. Incident times are synthetic.
Errors and limits
Six cases are too few for strong conclusions.
Exercises
- EX057 — Lab medians. Give the two lab medians. Rubric: 3 = acknowledge 7; contain 45.5; 2 = one; 1 = vague; 0 = none.
- EX058 — Why medians. Why use medians? Rubric: 3 = resist long outliers; 2 = partial; 1 = vague; 0 = none.
- EX059 — Bad metric. Name a metric that drives bad behaviour. Rubric: 3 = closed alerts per analyst; 2 = partial; 1 = vague; 0 = none.
- EX060 — I4 fix. What change followed the I4 outlier? Rubric: 3 = 15-min shift overlap; 2 = partial; 1 = vague; 0 = none.
Synthesis
Measure what matters, use medians, avoid perverse metrics and explain outliers.
Continue studying — chapter 15 of 18
← Chapter 14 (draft) · Course chapter index · Chapter 16 (draft) →