◇ OpenCyber Academy
Engineering · Chapter 24 · English

Chapter 24 — Security operations and blue team: methods I

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 24 — Security operations and blue team: methods I

Lesson map

Lesson 24 published. Map, example, EX070.

flowchart TD
  N0["Learning outcomes"]
  N1["24.1 Threshold rules"]
  N2["24.2 Correlation rules"]
  N3["Lab — comparing rule variants"]
  N4["24.3 Reading the result"]
  N5["24.4 Purpose decides"]
  N6["24.5 Labels"]
  N7["24.6 Small samples"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX070"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["24.1 Threshold rules"]
  N2["24.2 Correlation rules"]
  N3["Lab — comparing rule variants"]
  N4["24.3 Reading the result"]
  N5["24.4 Purpose decides"]
  N6["24.5 Labels"]
  N7["24.6 Small samples"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX070"]

[Learning outcomes]
    |
    v
[24.1 Threshold rules]
    |
    v
[24.2 Correlation rules]
    |
    v
[Lab — comparing rule variants]
    |
    v
[24.3 Reading the result]
    |
    v
[24.4 Purpose decides]
    |
    v
[24.5 Labels]
    |
    v
[24.6 Small samples]
    |
    v
[Worked example] --> [Exercise EX070]

Learning outcomes

You will (1) design threshold and correlation detections; (2) evaluate rules against labelled windows; (3) compare rule variants by true and false positives; (4) choose variants by operational purpose.

24.1 Threshold rules

A threshold rule fires when a count in a window crosses a limit: at least 5 distinct users failing from one /24 in 10 minutes.

Counting distinct users rather than total failures targets the stuffing pattern: many accounts, few tries each. A single customer failing ten times would not trip the rule, which is intended. The window and limit are parameters to tune, and the rule's documentation records the values chosen and the evidence that led to them. Five in ten minutes is the starting point.

24.2 Correlation rules

A correlation rule requires several conditions, possibly from different sources: failures followed by a success, or a success followed by an address change.

Correlation adds meaning by joining steps that are weak alone. Failures happen constantly; a success alone is normal; failures followed by a success from the same network suggest a guessed password. An address change after a new-network login is more suspicious than either event separately. Such rules need sources with aligned times and shared fields. Clock offsets between sources must be small.

Lab — comparing rule variants

"""Evaluate a Sigma-style burst rule over synthetic 10-minute windows with labels."""
windows = [  # (window, distinct users failed, any success, label)
    ("09:50", 2, True, "benign"), ("10:00", 7, True, "attack"), ("10:10", 6, False, "attack"),
    ("10:20", 1, False, "benign"), ("11:00", 9, False, "benign"),   # load test
    ("13:30", 5, True, "attack"), ("14:00", 3, True, "benign"), ("15:00", 6, False, "benign")]
def rule(users, success, threshold): return users >= threshold and success
for t in (5, 6):
    tp = sum(rule(u, s, t) and l == "attack" for _, u, s, l in windows)
    fp = sum(rule(u, s, t) and l == "benign" for _, u, s, l in windows)
    fn = sum(not rule(u, s, t) and l == "attack" for _, u, s, l in windows)
    print(f"threshold={t} with-success: TP={tp} FP={fp} FN={fn}")
tp = sum(u >= 5 and l == "attack" for _, u, s, l in windows); fp = sum(u >= 5 and l == "benign" for _, u, s, l in windows)
print(f"threshold=5 no success clause: TP={tp} FP={fp} FN={3-tp}")

Run with python3 lab_rule.py. Verified output:

threshold=5 with-success: TP=2 FP=0 FN=1
threshold=6 with-success: TP=1 FP=0 FN=2
threshold=5 no success clause: TP=3 FP=2 FN=0

24.3 Reading the result

With the success clause and threshold 5, the rule catches 2 of 3 attack windows with no false alerts; at threshold 6 it catches only 1. Without the success clause, threshold 5 catches all 3 attack windows but adds 2 false alerts — the 11:00 load test and a 15:00 window.

The figures show the trade-off clearly. Requiring a success makes the rule precise but costs one attack window, an attempt that never succeeded. Dropping the clause catches every attempt but adds noise, including a load test that a signed-tool exception could remove. Raising the threshold to six loses another real window without removing any false alerts, since there were none.

24.4 Purpose decides

The no-success variant detects attempts; the success variant detects likely compromise. Lindqvist uses both: attempts go to a daytime queue for rate-limit tuning; likely compromise pages the night shift.

Neither variant is simply better; each answers a different question. Attempts matter for tuning defences, such as rate limits, and can wait for daytime. Likely compromise needs action within minutes, so it justifies waking someone. Splitting the rule by purpose gives each the right urgency instead of forcing one compromise threshold on both needs.

24.5 Labels

Evaluation needs labelled windows. Labels come from investigated incidents, authorised tests and analyst review of samples.

Each label source has weaknesses. Investigated incidents are reliable but few; authorised tests are clean but may not look like real attacks; analyst reviews are plentiful but can be wrong. Recording where each label came from lets evaluators weight them and check whether results depend heavily on one kind of label. Beside each of her eight windows, Amara writes which source supplied its label and who confirmed it, so a reviewer can challenge any single label.

24.6 Small samples

Eight windows illustrate the method only; real evaluation uses weeks of data.

With eight windows, a single mislabelled one changes the results dramatically. Weeks of data give hundreds of windows, enough to estimate rates with useful confidence and to see variation between weekdays and weekends. The small set remains valuable for teaching and for unit tests, where every window can be checked by hand. Amara states the sample size beside every figure she reports, so nobody mistakes the teaching example for a measured performance of the rule in production.

Rule naming and severity

Each variant gets a clear name and severity. 'Portal stuffing attempts' is low and queued; 'Portal possible takeover' is high and pages. Names tell analysts what to expect before they open the alert, which speeds triage and reduces confusion between similar rules.

Worked example

Amara presents the results to Paolo as a table of variants, with a recommendation: page on "threshold 5 with success", queue "threshold 5 without success" for daytime review. The missed 10:10 attack window had no success, so the paging rule correctly stayed quiet — the attacker had not yet got in. This moves the paging threshold back from the 6 that the earlier tuning loop settled on to 5. That loop tuned the failures-only rule, where raising the threshold was the only way to cut false alerts. Once the success clause is added, it removes the false alerts by itself: on the lab windows, threshold 5 with a success has no false alerts and catches two attack windows, while 6 catches only one. Keeping 6 would lose detection and gain nothing, so R-07's threshold is amended to 5 through change control.

Synthetic case. Windows and labels come from the lab script.

Errors and limits

Labels themselves can be wrong; review disputed ones.

Exercises

Synthesis

Design rule variants for distinct purposes and choose them by measured results on labelled data.

Continue studying — chapter 24 of 50

← Chapter 23 (draft) · Course chapter index · Chapter 25 (draft) →