Chapter 23 — Security operations and blue team: evaluation
Lesson map
Lesson 23 published. Map, example, EX067.
flowchart TD
N0["Learning outcomes"]
N1["23.1 Why evaluate"]
N2["23.2 Measures"]
N3["Lab — monthly rule quality"]
N4["23.3 Reading the result"]
N5["23.4 Actions"]
N6["23.5 Small numbers"]
N7["23.6 Missed detections"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX067"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["23.1 Why evaluate"]
N2["23.2 Measures"]
N3["Lab — monthly rule quality"]
N4["23.3 Reading the result"]
N5["23.4 Actions"]
N6["23.5 Small numbers"]
N7["23.6 Missed detections"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX067"]
[Learning outcomes]
|
v
[23.1 Why evaluate]
|
v
[23.2 Measures]
|
v
[Lab — monthly rule quality]
|
v
[23.3 Reading the result]
|
v
[23.4 Actions]
|
v
[23.5 Small numbers]
|
v
[23.6 Missed detections]
|
v
[Worked example] --> [Exercise EX067]
Learning outcomes
You will (1) evaluate detection rules with precision and recall from case outcomes; (2) identify rules needing tuning or redesign; (3) consider missed detections; (4) report evaluation results.
23.1 Why evaluate
Rules drift as clients change. Monthly evaluation shows which rules work, which waste analyst time and which miss attacks.
BrightSmile alone shows how clients change: new personal accounts, a VPN, a new card terminal and a dentist's laptop, all within a few months. Each change can make a rule noisier or blind, for instance when the reception account no longer exists for the out-of-hours rule to watch. Monthly evaluation catches such drift before it shows up as a missed incident.
23.2 Measures
From closed cases, each alert is a true positive (real issue) or false positive. Missed detections (false negatives) come from incidents found other ways, such as client reports. Precision is TP ÷ (TP + FP); recall is TP ÷ (TP + FN).
The two measures answer different questions about the same rule. Precision asks how much of an analyst's time on a rule's alerts was well spent; recall asks how many real problems the rule caught. A rule can score well on one and badly on the other, and the AV-disabled rule shows the worst case, with both low.
Lab — monthly rule quality
"""Month-end detection quality per rule from closed-alert labels (synthetic)."""
rules = {"RDP brute force": (12, 3, 0), "new local admin": (4, 6, 0), "rename spike": (2, 1, 0), "AV disabled": (1, 9, 1)}
for name, (tp, fp, fn) in rules.items():
prec = tp / (tp + fp)
rec = tp / (tp + fn) if tp + fn else float("nan")
print(f"{name:16} TP={tp:2} FP={fp:2} FN={fn} precision={prec:.2f} recall={rec:.2f}")
Run with python3 lab_quality.py. Verified output:
RDP brute force TP=12 FP= 3 FN=0 precision=0.80 recall=1.00
new local admin TP= 4 FP= 6 FN=0 precision=0.40 recall=1.00
rename spike TP= 2 FP= 1 FN=0 precision=0.67 recall=1.00
AV disabled TP= 1 FP= 9 FN=1 precision=0.10 recall=0.50
23.3 Reading the result
The remote-logon brute-force rule has precision 0.80 and recall 1.00. New local admin has precision 0.40 — many legitimate account changes. Rename spike has 0.67. The AV-disabled rule has precision 0.10 and missed one real case (recall 0.50), the weakest rule.
Mei verifies the rates by hand: 12 of 15 brute-force alerts were real, giving 0.80; 4 of 10 new-admin alerts, giving 0.40; 2 of 3 rename alerts, giving 0.67; and 1 of 10 AV alerts, giving 0.10. For AV-disabled, one caught and one missed real case gives recall of 1 out of 2, which is 0.50. The brute-force rule is both precise and complete this month, so it needs no change and serves as the model for the others.
23.4 Actions
AV-disabled is redesigned (the ten-minute change from chapter 19, plus a check for tamper-protection events). New local admin gets the contractor-window exception from chapter 16.
Each action targets the cause found in the closed cases. Nine of the ten AV alerts were installers briefly pausing protection, so requiring ten minutes removes them, while the tamper-protection check aims at the missed case. For the new-admin rule, legitimate contractor work explains much of the noise, so the support-window exception addresses that directly. Each action has an owner in Mei's table, and next month's evaluation shows whether the changed rules actually improved.
23.5 Small numbers
A rule with three alerts a month gives unstable percentages; Mei reports counts beside rates.
The rename-spike rule shows why: two real alerts and one false give 0.67, but one more false alarm next month would drop it to 0.50 with nothing really changed. Printing "2 of 3" beside the rate lets Raj see that the change rests on a single alert, so he does not redesign a rule because of chance. Counts also show volume, which rates hide: a rule with 15 alerts and one with 3 can share a rate while costing analysts very different amounts of time.
23.6 Missed detections
The missed AV case was found by the client noticing pop-ups; the evaluation asks why the rule did not fire — the event type had changed after an antivirus update.
The cause was outside the rule itself: an antivirus update renamed the event the rule was looking for, so the rule kept running and kept matching nothing. Mei proposes a check that each rule's source event still arrives, similar to the data-gap monitoring of chapter 12, so a renamed event shows up as a drop instead of as silence.
Sharing results
A short version of the evaluation goes to clients in their monthly report: which alerts were real, which were noise, and what Copperleaf changed. Clients see that tuning is continuous work, not a one-off setup, and are more willing to report changes that affect rules.
Worked example
Mei's monthly evaluation table lists each rule's counts, rates, actions and owner. Raj presents it to the team and uses it to decide where engineering time goes next month.
Synthetic case. Counts come from the lab script.
Errors and limits
Labels depend on analysts' closure decisions, which may be wrong.
Exercises
- EX067 — Formulas. Write the formulas for precision and recall. Rubric: 3 = TP/(TP+FP); TP/(TP+FN); 2 = one; 1 = wrong; 0 = none.
- EX068 — Weakest rule. Which rule is weakest and why? Rubric: 3 = AV disabled: precision 0.10, recall 0.50; 2 = partial; 1 = other; 0 = none.
- EX069 — Miss cause. Why did the AV rule miss a real case? Rubric: 3 = event type changed after AV update; 2 = partial; 1 = unknown; 0 = none.
Synthesis
Regular evaluation with precision, recall and counts directs tuning and reveals silent rule failures.
Continue studying — chapter 23 of 30
← Chapter 22 (draft) · Course chapter index · Chapter 24 (draft) →