OpenCyber Academy
Doctoral research · Chapter 2 · English
Course code: OCA-199Chapter 02: OCA-199-C002

Chapter 2 — Security operations and blue team: research frontier

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 02 published. Map, example, EX008.

flowchart TD
  N0["Learning outcomes"]
  N1["2.1 The base-rate problem"]
  N2["Lab — precision under different base rates"]
  N3["2.2 Reading the result"]
  N4["2.3 Alert fatigue"]
  N5["2.4 Machine learning caution"]
  N6["2.5 Human and organisational factors"]
  N7["2.6 Nadia's position"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX008"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["2.1 The base-rate problem"]
  N2["Lab — precision under different base rates"]
  N3["2.2 Reading the result"]
  N4["2.3 Alert fatigue"]
  N5["2.4 Machine learning caution"]
  N6["2.5 Human and organisational factors"]
  N7["2.6 Nadia's position"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX008"]

[Learning outcomes]
    |
    v
[2.1 The base-rate problem]
    |
    v
[Lab — precision under different base rates]
    |
    v
[2.2 Reading the result]
    |
    v
[2.3 Alert fatigue]
    |
    v
[2.4 Machine learning caution]
    |
    v
[2.5 Human and organisational factors]
    |
    v
[2.6 Nadia's position]
    |
    v
[Worked example] --> [Exercise EX008]

Learning outcomes

You will (1) map open problems in security operations research; (2) explain the base-rate problem for alerting; (3) relate alert fatigue research to triage design; (4) position a thesis on the frontier.

2.1 The base-rate problem

Axelsson (2000) showed that because genuine intrusions are rare among the events a detector sees, even a small false-positive rate produces mostly false alarms. Precision depends on prevalence as well as on the detector. For Nadia's study, the base-rate argument is the motivation: if rules fire mostly on benign activity, the analyst's decision at triage decides whether a SOC is useful. Her experiment therefore measures what happens at that second stage, where the alerts that detection could not filter must be sorted by people.

Lab — precision under different base rates

"""Base-rate effect on alert precision (Axelsson 2000), synthetic figures."""
def ppv(tpr, fpr, prevalence):
    tp = tpr * prevalence
    fp = fpr * (1 - prevalence)
    return tp / (tp + fp)
for prev in (1e-2, 1e-3, 1e-4):
    for fpr in (1e-2, 1e-3):
        print(f"prevalence={prev:.0e} TPR=0.95 FPR={fpr:.0e} -> precision={ppv(0.95, fpr, prev):.3f}")

Run with python3 lab_baserate.py. Verified output:

prevalence=1e-02 TPR=0.95 FPR=1e-02 -> precision=0.490
prevalence=1e-02 TPR=0.95 FPR=1e-03 -> precision=0.906
prevalence=1e-03 TPR=0.95 FPR=1e-02 -> precision=0.087
prevalence=1e-03 TPR=0.95 FPR=1e-03 -> precision=0.487
prevalence=1e-04 TPR=0.95 FPR=1e-02 -> precision=0.009
prevalence=1e-04 TPR=0.95 FPR=1e-03 -> precision=0.087

2.2 Reading the result

With 95% true-positive rate and 1% false-positive rate, precision is 0.490 at a prevalence of one in a hundred, 0.087 at one in a thousand and 0.009 at one in ten thousand. Cutting the false-positive rate to 0.1% lifts the one-in-a-thousand case only to 0.487. Rare-event detection needs very low false-positive rates or a second stage, such as analyst triage, to be useful. Nadia uses the lab to set expectations for her corpus. Because real alert streams are dominated by benign triggers, her corpus includes many of them deliberately, so that triage decisions are tested on the alerts analysts most often see.

2.3 Alert fatigue

Alahmadi, Axon and Martinovic (2022) interviewed SOC practitioners and found that many alarms called "false positives" are benign triggers: the rule matched real behaviour that was legitimate in context. They propose that alarms be Reliable, Explainable, Analytical, Contextual and Transferable. Hassan et al. (2019) used provenance graphs to rank alerts and reduce fatigue. The benign-trigger finding is the starting point of Nadia's project: if many alarms are correct matches on legitimate activity, analysts need context to recognise legitimacy quickly. Her enriched cards test the Contextual property directly. Ranking methods like Hassan et al.'s reduce how many alerts reach analysts, which is complementary to her question rather than a competitor.

2.4 Machine learning caution

Sommer and Paxson (2010) argued that anomaly detection suits closed worlds poorly: attacks are rare, benign variety is huge, and errors are costly to explain. Their critique still frames how frontier work must be evaluated. Nadia's study avoids anomaly detection altogether; her rules are explicit Sigma conditions. The critique still shapes her design, because it warns that a method evaluated on convenient data can fail in operation. That is why she reports realism as a limitation and plans an operational pilot as the test that could falsify her claim.

2.5 Human and organisational factors

Sundaramurthy et al. (2014) embedded researchers in SOCs and found tacit knowledge and tooling friction central. Kokulu et al. (2019) stressed mismatches between management metrics and analyst reality. These studies remind Nadia that triage happens inside an organisation, with habits and tools that a laboratory session removes. Her pilot found exactly such friction: analysts did not see a field placed at the bottom of the card. A senior analyst's later comment that the raw command line was missing came from the same kind of tacit knowledge.

2.6 Nadia's position

Nadia's thesis sits between detection and triage: it tests whether adding context to alerts, as the "Contextual" property suggests, improves analysts' decisions on benign triggers without hiding true attacks. The position makes her claim specific. She does not test whether context improves detection, nor whether it shortens incidents; she tests whether analysts who see context reach the right disposition more often. Her worry from the start was the second half of the sentence: that context might also lead analysts to dismiss a true attack, which is why harms are measured alongside accuracy.

Worked example

Nadia's literature map has three columns — detector-level, alert-level and analyst-level work — and places her study in the alert-level column, citing Alahmadi et al. for motivation and Axelsson for why triage, not detection alone, decides usefulness.

Synthetic case. Figures come from the lab script.

Errors and limits

Frontier claims age quickly; Nadia dates her review and records search terms so it can be updated.

Exercises

Synthesis

The frontier is shaped by base rates, alarm quality and human factors; a thesis must place itself precisely among them.

Continue studying — chapter 02 of 08

← Chapter 01 (draft) · Course chapter index · Chapter 03 (draft) →

Worked answers

EX008. Since real intrusions are rare among events, even a tiny false-positive rate means most alerts are false.

EX009. 0.087 with a 1% false-positive rate and 0.487 with 0.1%.

EX010. A benign trigger is a correct match on real behaviour that happens to be legitimate in context; a false positive in the strict sense is a match on behaviour the rule did not intend.

EX011. Reliable, Explainable, Analytical, Contextual and Transferable.

EX012. Attacks are rare, normal behaviour is extremely varied and errors are expensive to explain, so anomaly detection fits poorly without careful evaluation.

EX013. At the alert and triage level: it tests whether contextual information helps analysts dismiss benign triggers while still catching real attacks.

EX014. She dates the review and keeps the exact search terms so it can be rerun.