Chapter 19 — Detection-to-response monitoring and feedback

Chapter 19 — Detection-to-response monitoring and feedback

Lesson map

Lesson 19 published. Map, example, EX055.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["19.1 From rule to runbook"]
  N2["19.2 Outcome labels"]
  N3["19.3 Coverage and misses"]
  N4["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4
  N4 --> EX["Worked example"]
  EX --> Q["Exercise EX055"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["19.1 From rule to runbook"]
  N2["19.2 Outcome labels"]
  N3["19.3 Coverage and misses"]
  N4["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX055"]

[Learning outcomes and connection]
    |
    v
[19.1 From rule to runbook]
    |
    v
[19.2 Outcome labels]
    |
    v
[19.3 Coverage and misses]
    |
    v
[Chapter synthesis]
    |
    v
[Worked example] --> [Exercise EX055]

Learning outcomes and connection

You will be able to (1) connect each detection rule to a response action and an owner; (2) record case outcomes back against the detections that raised them; and (3) use that feedback to tune, retire or create detections. Chapter 13 set thresholds; Chapter 18 built actions; this chapter closes the loop between them.

19.1 From rule to runbook

Every detection that can open a case should point to a runbook: what the analyst checks first, what evidence to preserve, which containment action applies and who approves it. A rule without a runbook produces inconsistent triage; a runbook without a rule depends on luck. Fernmoor keeps a detection register: rule id, scenario (Chapter 13), data source (Chapter 12), runbook, owner, last tuned date and quality figures. Fernmoor's register includes rules such as R-07 for impossible travel and R-12 for a new MFA method off-hours.

19.2 Outcome labels

When a case closes, the analyst labels the triggering alert: true positive (malicious or policy-violating activity), benign true positive (the rule worked, the activity was authorized), false positive (the rule matched something it should not), or undetermined. These labels, collected over months, show which rules earn their cost. The CSF 2.0 Detect function includes continuous monitoring and analysis of adverse events; feedback labels are how monitoring improves. Fernmoor's Q2 register shows why these labels matter. Rule R-19 for bulk file reads produced 120 alerts, mostly benign true positives from the nightly backup account; treating them as false positives would have led to removing a rule that does what it should. The labels instead pointed to a narrow exclusion for the backup window.

19.3 Coverage and misses

Also record cases that no detection found: reports from residents, suppliers or staff. Each miss is a candidate for a new detection or a data-source gap. Mapping cases to ATT&CK techniques helps reveal which techniques you detect and which you only learn about by accident. In Fernmoor's Q2 register, two cases came from staff reports, a phished account and an insider spreadsheet. Both were added to the backlog as detection ideas with owners, and the register now shows them against the techniques they used. Over several quarters, these misses show where Fernmoor relies on staff alertness instead of detection.

19.4 Worked example (synthetic): a quarter's register

Fictional. Q2 detection register excerpt:

Decisions: R-07 — add known corporate VPN egress as enrichment to cut benign TPs; R-12 — keep, high value; R-19 — most benign TPs are the nightly backup account; exclude it only for its scheduled window and host, and alert if it reads outside; R-23 — tune threshold upward and add sequential-id detection. Two cases came from staff reports (a phished account and an insider spreadsheet); both added detection ideas to the backlog with owners.

19.5 Common errors and limits

• Treating benign true positives as false positives, which leads to removing good rules.
• Broad exclusions ("ignore the backup account") that create blind spots attackers can use.
• Never labelling outcomes, so no rule is ever improved.
• Counting alerts as a productivity measure.

Fernmoor's register avoids these by design. R-19's benign alerts from the backup account were labelled correctly, the exclusion covered only the scheduled window and host, with an alert if the account read outside it, every closed case has an outcome label, and the register reports quality per rule rather than alert counts. R-12 was kept despite few alerts because all were meaningful.

Exercises — chapter 19

Attempt each exercise before opening the worked results. Every case, organization, person, address and number in an exercise is synthetic unless a cited public source is named.

EX055 — Precision. Steps: compute the share of R-07 and R-19 alerts that were true positives and interpret. Rubric: 3 = R-07 4/52 ≈ 7.7%, R-19 1/120 ≈ 0.8%, with interpretation that benign TPs dominate and enrichment helps; 2 = numbers only; 1 = wrong; 0 = none.

EX056 — Safe exclusion. Steps: write the R-19 exclusion precisely. Rubric: 3 = account, host, time window, share; alert on any deviation; review date; 2 = missing deviation alert; 1 = broad; 0 = none.

EX057 — From miss to rule. Steps: for the insider spreadsheet case found by a staff report, propose a detection with its data source and runbook first step. Rubric: 3 = specific logic (e.g., large export from billing DB to personal email/external domain), source, threshold, runbook step involving HR/privacy; 2 = vague logic; 1 = idea only; 0 = none.

Chapter synthesis

Link every detection to a runbook and owner, label outcomes at closure, record cases no rule found, and tune with narrow, monitored exclusions. The detection register is how the response service learns.

Continue studying — chapter 19 of 30

← Chapter 18 · Course chapter index · Chapter 20 →

Continue to the next authored lesson.