Chapter 7 — Measuring and tuning detections
Lesson map
Lesson 07 published. Map, example, EX031.
flowchart TD
N0["Learning outcomes"]
N1["7.1 Precision"]
N2["Lab — rule precision"]
N3["7.2 Lab result"]
N4["7.3 Base rate"]
N5["7.4 Tuning"]
N6["7.5 Recording"]
N7["Validation alongside precision"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX031"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["7.1 Precision"]
N2["Lab — rule precision"]
N3["7.2 Lab result"]
N4["7.3 Base rate"]
N5["7.4 Tuning"]
N6["7.5 Recording"]
N7["Validation alongside precision"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX031"]
[Learning outcomes]
|
v
[7.1 Precision]
|
v
[Lab — rule precision]
|
v
[7.2 Lab result]
|
v
[7.3 Base rate]
|
v
[7.4 Tuning]
|
v
[7.5 Recording]
|
v
[Validation alongside precision]
|
v
[Worked example] --> [Exercise EX031]
Learning outcomes
You will (1) measure rule precision from triage outcomes; (2) tune noisy rules safely; (3) understand the base-rate problem; (4) record tuning decisions.
7.1 Precision
Precision is the share of alerts that were real. Tom marks each closed alert real or benign so precision can be calculated. Tom records the outcome of each alert in the triage note, so precision can be calculated for each rule without extra work. A benign alert is not a failure in itself; rules need some false alarms to catch real attacks. Precision shows which rules waste the most analyst time, so tuning can focus there. Tom's records also show real cases a rule caught, which tuning must keep catching.
Lab — rule precision
"""Precision of each rule over one month of synthetic triage outcomes."""
rules = {"VPN login from new country": (40, 3), "Admin group change": (12, 4),
"AV detection on server": (6, 2), "Macro document opened": (95, 1)}
for name, (alerts, real) in sorted(rules.items(), key=lambda r: r[1][1] / r[1][0], reverse=True):
print(f"{name:28} alerts={alerts:3} real={real} precision={real / alerts:.2f}")
Run with python3 lab_precision.py. Verified output:
Admin group change alerts= 12 real=4 precision=0.33
AV detection on server alerts= 6 real=2 precision=0.33
VPN login from new country alerts= 40 real=3 precision=0.07
Macro document opened alerts= 95 real=1 precision=0.01
7.2 Lab result
Admin group change and AV detection on server each have precision 0.33; new-country VPN login 0.07 (3 real of 40); macro document opened 0.01 (1 of 95). Precision is very low for the macro and VPN rules, so most of their alerts are benign. The macro rule fired 95 times for one real case, which wastes analyst time and teaches staff to ignore it. Tom and Priya choose these two rules for tuning, since they cause most of the triage work. The other rules are kept as they are and reviewed again the following quarter.
7.3 Base rate
Axelsson (2000) showed that rare attacks make even good detectors produce mostly false alarms. The macro rule fires on any document with macros, and Fenwick's finance team uses many. Applied to Fenwick, the base rate explains why the macro rule's precision is so low: real malicious macros are rare, while finance staff open legitimate macro documents every day. Even a rule that catches every malicious macro will mostly flag legitimate ones. Tuning must reduce the benign triggers without losing the rare real case.
7.4 Tuning
The macro rule is narrowed to documents from outside email that start a new process. The VPN rule now checks the travel calendar first. Each change targets a known cause of benign alerts. Finance documents usually come from inside Fenwick and do not start new processes, so narrowing the macro rule removes most of them. Checking the travel calendar removes alerts for staff known to be abroad, like the sales manager at the trade fair. After tuning, the macro rule fired nine times in a month with one real case, and the VPN rule eight times with three real.
7.5 Recording
Each change has a reason, expected effect and review date; real cases are checked to ensure they would still be caught. Recording each change prevents tuning from quietly removing detection. Before a tuned rule goes live, Tom replays the real cases it caught earlier and checks that it still fires on them. The review date ensures that the tuning is checked again after a month of use. If precision does not improve as expected, the change is reconsidered. The record is part of the rule documentation, which the provider later peer-reviewed.
Validation alongside precision
Precision says how often alerts are right; validation says whether rules fire at all. Priya combines both: monthly precision from triage marks, and twice-yearly announced tests that confirm critical rules still fire. When the VPN product's log format changed in a patch, the validation test caught that the new-country rule had silently stopped firing, something precision figures could never show because a rule that never fires has no false alarms. The format mapping was fixed the same day.
Analyst input
Tom suggests tuning ideas from his triage experience each month, and they are tested before use.
Worked example
After tuning, the macro rule fires 9 times in a month with 1 real case; the VPN rule fires 8 times with 3 real. Tom's triage time falls by about four hours a week, which he spends improving runbooks.
Synthetic case. Counts are synthetic.
Errors and limits
Precision does not show missed attacks; validation tests (chapter 6) help there.
Exercises
- EX031 — Lab precision. Give the lowest and highest precision in the lab. Rubric: 3 = macro 0.01; admin/AV 0.33; 2 = one; 1 = vague; 0 = none.
- EX032 — Base rate. Explain the base-rate problem. Rubric: 3 = rare attacks; mostly false alarms; 2 = partial; 1 = vague; 0 = none.
- EX033 — Macro tuning. How was the macro rule tuned? Rubric: 3 = external email docs starting new process; 2 = partial; 1 = vague; 0 = none.
- EX034 — Post-tuning. What were the results after tuning? Rubric: 3 = macro 9 alerts 1 real; VPN 8 alerts 3 real; 2 = partial; 1 = vague; 0 = none.
- EX035 — Precision limit. What does precision not show? Rubric: 3 = missed attacks; 2 = partial; 1 = vague; 0 = none.
Synthesis
Measure precision, tune with care and keep real cases detectable.
Continue studying — chapter 07 of 12
← Chapter 06 (draft) · Course chapter index · Chapter 08 (draft) →