Chapter 13 — Tuning and false positives | Technical Security operations and blue team

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 13 — Tuning and false positives

Lesson map

Lesson 13 published. Map, example, EX049.

flowchart TD
  N0["Learning outcomes"]
  N1["13.1 Definitions"]
  N2["13.2 Measuring"]
  N3["13.3 Tuning options"]
  N4["13.4 The admin-tool rule"]
  N5["13.5 Recording"]
  N6["13.6 Watching after"]
  N7["13.7 Research context"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX049"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["13.1 Definitions"]
  N2["13.2 Measuring"]
  N3["13.3 Tuning options"]
  N4["13.4 The admin-tool rule"]
  N5["13.5 Recording"]
  N6["13.6 Watching after"]
  N7["13.7 Research context"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX049"]

[Learning outcomes]
    |
    v
[13.1 Definitions]
    |
    v
[13.2 Measuring]
    |
    v
[13.3 Tuning options]
    |
    v
[13.4 The admin-tool rule]
    |
    v
[13.5 Recording]
    |
    v
[13.6 Watching after]
    |
    v
[13.7 Research context]
    |
    v
[Worked example] --> [Exercise EX049]

Learning outcomes

You will (1) define false positives and false negatives; (2) measure a rule's usefulness; (3) tune rules without hiding real threats; (4) record tuning decisions.

13.1 Definitions

A false positive is an alert that is not a real problem; a false negative is a real problem that raised no alert. Axelsson (2000) showed that when real attacks are rare, even accurate detectors produce mostly false alarms; this is the base-rate problem. At Northgate, most alerts on most days are false positives, even from well-written rules, because real attacks are rare. The 'admin tool used' rule shows this: 220 alerts and two real cases. A false negative is harder to see, because nothing appears in the queue to count.

13.2 Measuring

For each rule Carlos counts alerts in a month and how many were real. The "admin tool used" rule fired 220 times; 2 were real. Two real cases out of 220 means about 1 in 110 alerts mattered, so analysts spent most effort on harmless activity. The analysts' marks on closed alerts, real, benign or unclear, supply these numbers. Without them, Carlos could not tell which rules need tuning first. Analysts who mark alerts carefully make this measurement possible, which is why marking is part of triage.

13.3 Tuning options

Exclude known safe sources, raise thresholds, add context such as time of day, or combine with another signal. Each option risks missing something. Excluding a source narrows what the rule sees, raising a threshold can let slow attacks through, and time-of-day context can miss an attacker working office hours. Carlos picks the option that removes the most noise while keeping the real cases, and records the risk he accepted. No option is free, so each tuning change states which kind of miss it accepts. Carlos discusses each trade-off with Aisha before applying a tuning change to a live rule.

13.4 The admin-tool rule

Most firings were IT staff using the tool from their own machines. Carlos excludes the IT admin workstations by name, keeping alerts for the tool used anywhere else. The real cases came from machines outside the IT admin workstations, so excluding those named machines removed the noise without losing them. Excluding all IT staff accounts instead would have been broader and could hide an attacker using a stolen IT account on another machine. The named workstations are listed in the tuning note, so a departed admin's machine can be spotted and removed.

13.5 Recording

The tuning note states the change, reason, expected effect and review date. For the admin-tool rule, the note says IT admin workstations were excluded by name because they produced most firings. It expects alerts to fall sharply while keeping cases from other machines, and it sets a quarterly review date. Anyone questioning the rule later can see why it changed. The tuning note is stored with the rule's documentation page in the SOC's knowledge base. Anyone reviewing the admin-tool rule months later can therefore find the reason for the exclusion quickly. The review date is also added to the team's calendar.

13.6 Watching after

In the next month the rule fires 14 times, 2 real. The exclusion list is reviewed quarterly so departed staff's machines are removed. Alerts fell from 220 to 14 a month, while the two real cases were still caught. Precision rose from about 1 in 110 to 1 in 7. The quarterly review of the exclusion list keeps it from growing stale as staff and machines change. The analysts' marks confirm the two real cases each month. Carlos keeps the monthly counts with the tuning note, so the next review starts from them.

13.7 Research context

Sommer and Paxson (2010) warned that detection systems which seem accurate in testing often struggle in real environments; local tuning is how SOCs cope. Northgate's experience fits this warning: the admin-tool rule looked reasonable on paper but produced 220 alerts in a real month. Only local measurement and tuning, based on the analysts' marks, made it useful. Rules shared through Sigma still need the same local tuning before they suit the council. The paper's point is that testing alone cannot guarantee real-world performance.

Feedback from analysts

Analysts mark each closed alert as real, benign or unclear. These marks give Carlos the numbers he needs for tuning, so marking honestly is part of every triage.

Review dates

Every tuning change gets a review date, and Carlos checks the rule's real-case count again on that date before keeping it.

Worked example

An analyst asks to silence the rule entirely. Carlos shows the two real cases came from outside the IT workstations, so the exclusion keeps them. Silencing would have missed both.

Synthetic case. Counts are synthetic.

Errors and limits

Every exclusion is a place an attacker could hide; keep them narrow and reviewed.

Exercises

EX049 — Base rate. What is the base-rate problem? Rubric: 3 = rare attacks mean mostly false alarms; 2 = partial; 1 = vague; 0 = none.
EX050 — Tuning change. How was the admin-tool rule tuned? Rubric: 3 = excluded named IT admin workstations; 2 = partial; 1 = vague; 0 = none.
EX051 — Result. What changed after tuning? Rubric: 3 = 220 to 14 alerts; 2 real still caught; 2 = partial; 1 = vague; 0 = none.
EX052 — Why not silence. Why not silence the rule? Rubric: 3 = would miss the 2 real cases; 2 = partial; 1 = vague; 0 = none.

Synthesis

Tune narrowly, record why, and check real cases are still caught.

Continue studying — chapter 13 of 18

← Chapter 12 (draft) · Course chapter index · Chapter 14 (draft) →