◇ OpenCyber Academy
Engineering · Chapter 7 · English

Chapter 7 — Probability and statistics

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 7 — Probability and statistics

Lesson map

Lesson 07 published. Map, example, EX019.

flowchart TD
  N0["Learning outcomes"]
  N1["7.1 Conditional probability"]
  N2["7.2 Base rates"]
  N3["Lab — Bayes for one rule"]
  N4["7.3 Reading the result"]
  N5["7.4 Summaries"]
  N6["7.5 Using statistics"]
  N7["Intervals for rates"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX019"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["7.1 Conditional probability"]
  N2["7.2 Base rates"]
  N3["Lab — Bayes for one rule"]
  N4["7.3 Reading the result"]
  N5["7.4 Summaries"]
  N6["7.5 Using statistics"]
  N7["Intervals for rates"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX019"]

[Learning outcomes]
    |
    v
[7.1 Conditional probability]
    |
    v
[7.2 Base rates]
    |
    v
[Lab — Bayes for one rule]
    |
    v
[7.3 Reading the result]
    |
    v
[7.4 Summaries]
    |
    v
[7.5 Using statistics]
    |
    v
[Intervals for rates]
    |
    v
[Worked example] --> [Exercise EX019]

Learning outcomes

You will (1) apply Bayes' rule to alert precision; (2) distinguish false-positive rate from precision; (3) summarise data with medians and percentiles; (4) recognise base-rate effects.

7.1 Conditional probability

Two questions are easy to confuse. "If an attack happens, how likely is an alert?" is the true-positive rate. "If an alert fires, how likely is an attack?" is precision. Bayes' rule connects them through the base rate.

Confusing them leads to bad decisions. A vendor may advertise that its detector catches 99% of attacks, which is the true-positive rate, while analysts experience something different: most alerts they open turn out to be benign. Bayes' rule shows how both can be true at once, because precision depends heavily on how rare attacks are among the events being judged.

7.2 Base rates

Axelsson (2000) showed that when attacks are rare among events, even a good detector produces mostly false alarms. Precision falls as prevalence falls.

Axelsson's argument applied this to intrusion detection systems at a time when vendors focused on detection rates. Suppose only one event in ten thousand is malicious; even a false-positive rate of one in a thousand produces ten false alarms for each real one. The practical lesson is that false-positive rates must be extremely low, or alerts must be filtered further before reaching people.

Lab — Bayes for one rule

"""Bayes' rule for one detection rule at Lindqvist Freight (synthetic figures)."""
p_attack = 0.002          # share of evaluated logon bursts that are credential stuffing
tpr, fpr = 0.90, 0.01
p_alert = tpr * p_attack + fpr * (1 - p_attack)
ppv = tpr * p_attack / p_alert
print(f"P(alert)={p_alert:.5f}  P(attack|alert)={ppv:.3f}")
print(f"per 100,000 bursts: true alerts={tpr*p_attack*1e5:.0f} false alerts={fpr*(1-p_attack)*1e5:.0f}")

Run with python3 lab_bayes.py. Verified output:

P(alert)=0.01178  P(attack|alert)=0.153
per 100,000 bursts: true alerts=180 false alerts=998

7.3 Reading the result

With 0.2% of logon bursts being credential stuffing, a 90% true-positive rate and a 1% false-positive rate, precision is 0.153: about one alert in seven is real. Per 100,000 bursts, 180 true and 998 false alerts are expected.

Seeing the counts makes the point vivid: 998 false alarms against 180 real ones. Halving the false-positive rate to 0.5% would cut false alarms to 499 and raise precision to about 0.27, so improving specificity pays off more than improving detection here. Amara checks the arithmetic by building the full table of attacks and non-attacks before trusting the formula's output.

7.4 Summaries

Triage times are skewed: most are quick, a few take hours. Medians and 90th percentiles describe them better than means.

A few long investigations can drag the mean far above what a typical alert takes, giving a misleading picture of daily work. The median says what a typical alert costs; the 90th percentile shows how long the slow ones run, which matters for staffing. Amara reports both, with the number of alerts in the sample, so readers can see how solid the figures are.

7.5 Using statistics

Amara uses these numbers to argue that the burst rule needs a second condition (a success in the burst) to raise precision before paging analysts at night.

Requiring a success within the burst changes the base rate: bursts with a success are a far smaller, more suspicious group, so a much larger share of alerts that fire on them are real. The trade-off is that attacks which fail entirely no longer page anyone at night. Amara proposes logging those to a daytime review queue so they are still seen.

Intervals for rates

An observed false-positive rate of 1% from 2,000 benign bursts has uncertainty: the true rate might be somewhat higher or lower. Amara reports rates with 95% intervals and notes the sample size, so readers do not treat 1% as exact. When samples are small, she says so and avoids decisions that depend on fine differences between rules. Paolo asks every rule proposal to include the sample on which its rates were measured.

Worked example

Paolo asks: "If we halve the false-positive rate, what happens?" Amara recomputes with 0.5%: precision rises to about 0.27. Halving false positives nearly doubles the share of real alerts, which is more useful than raising sensitivity from 90% to 95%.

Synthetic case. Rates come from the lab script; the 0.5% case is her recalculation.

Errors and limits

Rates estimated from small samples are uncertain; report intervals when possible.

Exercises

Synthesis

Bayes' rule and base rates explain why precision, not just detection rate, decides whether a rule is useful.

Continue studying — chapter 07 of 50

← Chapter 06 (draft) · Course chapter index · Chapter 08 (draft) →