Chapter 6 — Data collection and analysis
Lesson map
Lesson 06 published. Map, example, EX033.
flowchart TD
N0["Learning outcomes"]
N1["6.1 Collection"]
N2["6.2 Labelling results"]
N3["6.3 Checking allow-lists against ground truth"]
N4["Lab — allow-list replay"]
N5["6.4 Reading the result"]
N6["6.5 Handling the finding"]
N7["6.6 Overall results"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX033"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["6.1 Collection"]
N2["6.2 Labelling results"]
N3["6.3 Checking allow-lists against ground truth"]
N4["Lab — allow-list replay"]
N5["6.4 Reading the result"]
N6["6.5 Handling the finding"]
N7["6.6 Overall results"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX033"]
[Learning outcomes]
|
v
[6.1 Collection]
|
v
[6.2 Labelling results]
|
v
[6.3 Checking allow-lists against ground truth]
|
v
[Lab — allow-list replay]
|
v
[6.4 Reading the result]
|
v
[6.5 Handling the finding]
|
v
[6.6 Overall results]
|
v
[Worked example] --> [Exercise EX033]
Learning outcomes
You will (1) collect operational data under the approved protocol; (2) evaluate allow-lists against injected ground truth; (3) combine volume, labels and safety outcomes; (4) handle unexpected findings properly.
6.1 Collection
Weekly exports from the SIEM give alert counts per rule, pseudonymised at source. The labelling sample is drawn with the registered seed. Purple-team events are logged by the exercise lead with exact times.
Logging purple-team events independently of the SIEM is the key safeguard: if an allow-list suppresses an event, the SIEM has no record of it, but the exercise lead's log still shows the time and target. Drawing the labelling sample with the registered seed means the 200 alerts were chosen before anyone saw which were interesting.
6.2 Labelling results
Of the reweighted sample of alerts removed by allow-lists, an estimated 93% were benign triggers, 6% false positives and 1% unresolved. This supports the claim that tuning removed mostly legitimate activity.
Reweighting matters here because the sample over-represents high-severity alerts. Without it, the composition of the 20 high-severity alerts would count far more than their share of the population. The unresolved 1% is reported rather than assigned, because guessing those labels would make the benign share look more certain than it is. The 6% false positives point to rules that need fixing rather than allow-listing.
6.3 Checking allow-lists against ground truth
Allow-list entries suppress alerts for specific rule, host and account combinations. The concern is that an attacker using an allow-listed account on an allow-listed host would be hidden.
The concern follows from how allow-lists work: an entry that says "this account on this host is expected to do this" also tells the rule to ignore anyone who has taken over that account on that host. Checking entries against purple-team ground truth is the only way to see this, because a suppressed alert leaves no trace in the alert data.
Lab — allow-list replay
"""Apply an allow-list to synthetic labelled alerts; check for suppressed attacks."""
alerts = [
("ps-encoded", "SCADA-HIST01", "svc_backup", "benign"),
("ps-encoded", "SCADA-HIST01", "svc_backup", "benign"),
("ps-encoded", "FIN-PC07", "a.moss", "attack"),
("new-service", "PATCH-SRV2", "svc_patch", "benign"),
("new-service", "PATCH-SRV2", "svc_patch", "attack"),
("new-service", "HR-PC03", "j.ward", "benign"),
("rdp-logon", "JUMP01", "it.admin", "benign"),
("rdp-logon", "JUMP01", "it.admin", "benign"),
]
allow = {("ps-encoded", "SCADA-HIST01", "svc_backup"), ("new-service", "PATCH-SRV2", "svc_patch"), ("rdp-logon", "JUMP01", "it.admin")}
kept = [a for a in alerts if a[:3] not in allow]
suppressed = [a for a in alerts if a[:3] in allow]
print("alerts", len(alerts), "kept", len(kept), "suppressed", len(suppressed))
print("attacks kept", sum(a[3] == "attack" for a in kept), "attacks suppressed", sum(a[3] == "attack" for a in suppressed))
for a in suppressed:
if a[3] == "attack": print("SUPPRESSED ATTACK:", a[:3])
Run with python3 lab_allow.py. Verified output:
alerts 8 kept 2 suppressed 6
attacks kept 1 attacks suppressed 1
SUPPRESSED ATTACK: ('new-service', 'PATCH-SRV2', 'svc_patch')
6.4 Reading the result
Of 8 sample alerts, 6 are suppressed and 2 kept. One kept alert is an attack (FIN-PC07). One suppressed alert is an attack: a new service on PATCH-SRV2 by svc_patch — the purple team's simulated misuse of the patching account. The allow-list "new-service on PATCH-SRV2 by svc_patch" hides exactly this.
This is the case the study was designed to catch. The suppressed event is not a flaw in the rule but in the allow-list entry: by trusting every new service that svc_patch creates on PATCH-SRV2, it trusted the misuse as well. A volume-only evaluation would have counted that entry as a success.
6.5 Handling the finding
Per the registered stopping rule, the entry is reviewed within a day. Marcus Bell narrows it: new services by svc_patch on PATCH-SRV2 are allowed only when the service name matches the patch tool's signed naming pattern and occurs in the maintenance window. A re-replay shows the injected event now alerts.
The narrowed entry still removes the routine noise from legitimate patching, because real patch services match the signed naming pattern and run in the maintenance window. What it no longer trusts is the account alone. The re-replay confirms the fix on the injected case; the change log records the narrowing of entry AL-07 in week 6.
6.6 Overall results
Net reduction 136.5 alerts/week (chapter 4); 93% of removed alerts benign; 1 of 8 injected events suppressed before narrowing, 0 after.
The three results answer the question's three parts: volume fell, what was removed was mostly benign, and the one suppressed attack was found and fixed. Reporting them together prevents any one being quoted alone, such as the volume reduction without the suppressed event. The second result rests on the stratified labelling of 200 alerts, reweighted by stratum size; the third rests on the exercise lead's independent log of all eight injected events. Each has its own source file in the archive.
Worked example
Tariq's results table has three rows — volume, label composition, safety — each with estimate, uncertainty and source file, plus a note: "Safety finding led to narrowing of allow-list entry AL-07 in week 6."
Synthetic case. Alerts come from the lab script.
Errors and limits
Eight injected events are few; zero suppressions after narrowing does not prove no future suppression.
Exercises
- EX033 — Label composition. What share of removed alerts were benign triggers? Rubric: 3 = about 93%, reweighted; 2 = number only; 1 = wrong; 0 = none.
- EX034 — Allow-list risk. Why can allow-lists hide attacks? Rubric: 3 = attacker using allow-listed account/host combination is suppressed; 2 = partial; 1 = cannot; 0 = none.
- EX035 — Replay output. Summarise the lab output. Rubric: 3 = 8 alerts, 6 suppressed, 2 kept; 1 attack kept, 1 suppressed (PATCH-SRV2 svc_patch); 2 = partial; 1 = wrong; 0 = none.
- EX036 — Narrowing. How was entry AL-07 narrowed? Rubric: 3 = signed naming pattern plus maintenance window; re-replay alerts; 2 = partial; 1 = deleted rule; 0 = none.
- EX037 — Three outcomes. Report the three overall results. Rubric: 3 = −136.5/week, 93% benign, 1→0 suppressed injections; 2 = two; 1 = one; 0 = none.
- EX038 — Zero suppressions. Why does zero after narrowing not prove safety? Rubric: 3 = few injected events; future behaviours may differ; 2 = partial; 1 = proves it; 0 = none.
Synthesis
Measure volume, composition and safety together; when ground truth exposes a hole, fix it under the protocol and report it.
Continue studying — chapter 06 of 08
← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →