Chapter 6 — Data collection and analysis
Lesson map
Lesson 06 published. Map, example, EX033.
flowchart TD
N0["Learning outcomes"]
N1["6.1 Rollout"]
N2["Lab — Precision and recall"]
N3["6.2 Reading the result"]
N4["6.3 Examining the miss"]
N5["Weekly stability"]
N6["Analyst time"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX033"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["6.1 Rollout"]
N2["Lab — Precision and recall"]
N3["6.2 Reading the result"]
N4["6.3 Examining the miss"]
N5["Weekly stability"]
N6["Analyst time"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX033"]
[Learning outcomes]
|
v
[6.1 Rollout]
|
v
[Lab — Precision and recall]
|
v
[6.2 Reading the result]
|
v
[6.3 Examining the miss]
|
v
[Weekly stability]
|
v
[Analyst time]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX033]
Learning outcomes
You will (1) apply tuning estate-wide; (2) measure precision and recall; (3) analyse results.
6.1 Rollout
After the pilot, the revised procedure is applied to all sensors in week five, with each change in the register. Amara collects four weeks of post-tuning data.
The rollout extends tuning to both sensors in planned stages, using the revised procedure from the pilot. Each stage has the same review and logging, plus checks that alert volume and key detections behave as expected. Analysts are briefed before each stage, and any surprising drop in a rule's alerts is investigated before the next stage begins.
Lab — Precision and recall
"""Alert counts and recall on a labelled synthetic week, before and after tuning (stdlib only)."""
before = {"alerts": 4210, "true": 38, "labelled_true_events": 40}
after = {"alerts": 1180, "true": 37, "labelled_true_events": 40}
for k, d in (("before", before), ("after", after)):
print(f"{k:6} alerts {d['alerts']:5d} precision {d['true']/d['alerts']:.3f} recall {d['true']/d['labelled_true_events']:.3f}")
print(f"alert reduction {1-after['alerts']/before['alerts']:.1%}; true detections lost {before['true']-after['true']}")
Run with python3 lab_tuning.py. Verified output:
before alerts 4210 precision 0.009 recall 0.950
after alerts 1180 precision 0.031 recall 0.925
alert reduction 72.0%; true detections lost 1
6.2 Reading the result
Alerts fall 72 percent in raw terms. Precision rises from 0.009 to 0.031, meaning analysts now see about three actionable alerts per hundred instead of one. Recall falls from 0.950 to 0.925: one labelled event that previously alerted no longer does. Both success criteria are met, but the lost detection must be examined, not hidden.
She reports both volume reduction and detection retention with intervals.
She also notes which rules drove most of the reduction, because a few noisy rules often account for most volume.
6.3 Examining the miss
The missed event was an authorised test of unusual file transfer from a server that the backup exception now covers. The exception was narrower than in the pilot draft but still too broad in time. Amara proposes restricting the backup exception to the scheduled backup window, and checks on the labelled set that this restores the detection without much increase in alerts.
The miss is treated as a lesson, not hidden.
Its cause informs the review procedure.
Retesting confirms the fix.
Weekly stability
Averages can hide unstable weeks. Amara plots weekly precision and recall across the four post-tuning weeks and finds them stable, with recall at 0.925 in every week because the same labelled event is missed. Alerts per hundred gigabytes vary between about 185 and 194, a narrow band compared with the pre-tuning level above 510. This stability suggests the improvement is not an artefact of one quiet week. She also checks the four busiest rules after tuning to confirm none has become a new dominant source of noise, which would indicate a tuning change had shifted the problem rather than solved it.
Analyst time
The practical value of tuning is analyst time. Before tuning, analysts estimated spending about two minutes per alert they actually looked at, and they looked at only a fraction. After tuning, with roughly a quarter of the volume, they report looking at most alerts and closing them with notes. Amara records these estimates from short structured interviews, treating them as self-reported and approximate. Even so, they show the direction of benefit that matters to Corvane: attention is now spread over fewer alerts, so the actionable ones are more likely to be noticed.
Worked example
With the window restriction, recall returns to 0.950 and weekly alerts rise by only 40.
Synthetic case. Data are synthetic.
Errors and limits
Celebrating volume reduction while ignoring a lost detection would misrepresent the result.
Exercises
- EX033 — Precision meaning. Explain precision 0.031 in analyst terms. Rubric: 3 = about 3 per 100 actionable;; 2 = three percent;; 1 = vague;; 0 = none.
- EX034 — Recall change. What does recall 0.950 to 0.925 mean? Rubric: 3 = one fewer detected event;; 2 = lower;; 1 = vague;; 0 = none.
- EX035 — Miss cause. What caused the miss? Rubric: 3 = backup exception too broad in time;; 2 = exception;; 1 = vague;; 0 = none.
- EX036 — Fix test. How was the fix tested? Rubric: 3 = rerun on labelled set;; 2 = tested;; 1 = vague;; 0 = none.
- EX037 — Cost of fix. What did the fix cost in alerts? Rubric: 3 = about 40 per week;; 2 = small;; 1 = vague;; 0 = none.
- EX038 — Honest reporting. Why report the lost detection prominently? Rubric: 3 = integrity and improvement;; 2 = honesty;; 1 = vague;; 0 = none.
Synthesis
Tuning met its criteria, and analysing the one miss improved the procedure further.
Continue studying — chapter 06 of 08
← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →