Chapter 4 — Security operations and blue team: competing explanations
Lesson map
Lesson 04 published. Map, example, EX021.
flowchart TD
N0["Learning outcomes"]
N1["4.1 The observation"]
N2["4.2 Comparison group"]
N3["Lab — difference-in-differences"]
N4["4.3 Reading the result"]
N5["4.4 Testing specific alternatives"]
N6["4.5 Parallel trends"]
N7["4.6 Remaining uncertainty"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX021"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["4.1 The observation"]
N2["4.2 Comparison group"]
N3["Lab — difference-in-differences"]
N4["4.3 Reading the result"]
N5["4.4 Testing specific alternatives"]
N6["4.5 Parallel trends"]
N7["4.6 Remaining uncertainty"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX021"]
[Learning outcomes]
|
v
[4.1 The observation]
|
v
[4.2 Comparison group]
|
v
[Lab — difference-in-differences]
|
v
[4.3 Reading the result]
|
v
[4.4 Testing specific alternatives]
|
v
[4.5 Parallel trends]
|
v
[4.6 Remaining uncertainty]
|
v
[Worked example] --> [Exercise EX021]
Learning outcomes
You will (1) list competing explanations for an observed change; (2) use a comparison group to remove shared effects; (3) test specific alternatives with data; (4) state what remains uncertain.
4.1 The observation
After allow-lists were added in week 5, alerts from the ten tuned rules fell sharply. Tempting conclusion: tuning worked. But other things happened in weeks 5–8: a public holiday, a SIEM upgrade and a firewall log source migration.
The three events each threaten the conclusion differently. A holiday lowers activity, and so alerts, across the board. A SIEM upgrade could change how rules fire. A log-source migration could cut a rule's input. Any of them, happening in week 5, could produce a sharp fall in tuned rules without the allow-lists doing anything.
4.2 Comparison group
If the drop were due to the holiday or the SIEM upgrade, untuned rules should fall too. Difference-in-differences subtracts the control group's change from the tuned group's change.
The control rules are the reason Tariq can separate these effects at all. Without them, a fall in the tuned rules during a holiday week would be uninterpretable. With them, the holiday and the SIEM upgrade show up in both groups, and only the part of the fall unique to the tuned rules is attributed to tuning.
Lab — difference-in-differences
"""Difference-in-differences on synthetic weekly alert counts.
Tuned group: rules given allow-lists in week 5. Control group: untuned rules."""
tuned = [410, 395, 402, 420, 230, 241, 225, 236]
control = [300, 290, 305, 310, 262, 270, 258, 266]
def mean(x): return sum(x) / len(x)
t_pre, t_post = mean(tuned[:4]), mean(tuned[4:])
c_pre, c_post = mean(control[:4]), mean(control[4:])
print(f"tuned pre={t_pre:.1f} post={t_post:.1f} change={t_post-t_pre:.1f}")
print(f"control pre={c_pre:.1f} post={c_post:.1f} change={c_post-c_pre:.1f}")
print(f"difference-in-differences = {(t_post-t_pre)-(c_post-c_pre):.1f} alerts/week")
Run with python3 lab_did.py. Verified output:
tuned pre=406.8 post=233.0 change=-173.8
control pre=301.2 post=264.0 change=-37.2
difference-in-differences = -136.5 alerts/week
4.3 Reading the result
Tuned rules fell from 406.8 to 233.0 alerts per week (−173.8). Control rules also fell, from 301.2 to 264.0 (−37.2), showing a shared effect. The net estimate attributable to tuning is −136.5 alerts per week.
The printed changes are rounded to one decimal place; computed from the unrounded group means, the tuned change is −173.75 and the control change −37.25, which is why the net estimate is −136.5 rather than −136.6. Tariq reports the net estimate, not the raw fall, as the effect of tuning. The rounding note matters because an examiner checking the arithmetic by hand would otherwise suspect an error.
4.4 Testing specific alternatives
Log-source outage: ingestion counts per source show no gaps in weeks 5–8. Firewall migration: none of the tuned rules use firewall logs. Colleague tuning: the change log shows no other edits to the twenty rules. Holiday: affects both groups; absorbed by the comparison.
Each alternative was checked with its own evidence rather than argued away: ingestion counts for the outage, the rule definitions for the firewall migration and the change log for colleague tuning. Only the holiday needed the comparison group, because it affected both.
4.5 Parallel trends
Difference-in-differences assumes both groups would have moved alike without tuning. In weeks 1–4 their trends are similar (both roughly flat), which supports but does not prove the assumption.
The four baseline weeks show both groups roughly flat: tuned rules between 395 and 420 alerts a week, control rules between 290 and 310. If the tuned rules had been falling already, part of the post-week-5 drop would have been a continuing trend. Similar pre-period behaviour is the best evidence available that they would have stayed flat together.
4.6 Remaining uncertainty
Eight weeks is short; the estimate's interval is wide because weekly counts vary. Tariq reports the point estimate with a bootstrap interval in the dissertation.
The bootstrap interval reported in the dissertation is 98 to 171 alerts per week. It is wide, but it excludes zero and small effects, so the conclusion that tuning reduced volume holds even at the lower end. The width tells Pelham not to promise a precise saving to analysts. More weeks after tuning would narrow it, because the weekly counts vary by up to about 25 alerts from one week to the next, and eight weeks give only four post-tuning values per group.
Worked example
Tariq's alternatives table: each explanation, the evidence that would support it, the check run and the result. Only the tuning explanation survives all checks; the holiday explains the control group's fall.
Synthetic case. Counts come from the lab script.
Errors and limits
Comparison groups remove shared effects only; something affecting tuned rules alone would still bias the estimate.
Exercises
- EX021 — Alternatives list. Name three competing explanations for the drop. Rubric: 3 = holiday, SIEM upgrade, firewall migration (or outage/colleague tuning); 2 = two; 1 = one; 0 = none.
- EX022 — DiD logic. Explain difference-in-differences in one sentence. Rubric: 3 = subtract control change from treated change to remove shared effects; 2 = partial; 1 = vague; 0 = none.
- EX023 — Lab estimate. Give both group changes and the net estimate. Rubric: 3 = −173.8, −37.2, −136.5; 2 = two; 1 = one; 0 = none.
- EX024 — Specific checks. How was the firewall migration ruled out? Rubric: 3 = tuned rules do not use firewall logs; 2 = partial; 1 = assumed; 0 = none.
- EX025 — Parallel trends. What assumption does the method need, and what supports it? Rubric: 3 = parallel trends; similar flat pre-period; 2 = assumption only; 1 = vague; 0 = none.
- EX026 — Residual bias. What could still bias the estimate? Rubric: 3 = something affecting tuned rules only; 2 = partial; 1 = vague; 0 = none.
Synthesis
Before crediting an intervention, list rivals, subtract shared effects and test the rest one by one.
Continue studying — chapter 04 of 08
← Chapter 03 (draft) · Course chapter index · Chapter 05 (draft) →