Chapter 07 — Probability and statistics
Lesson map
Lesson 07 published. Map, example, EX013.
flowchart TD
N0["Learning outcomes and connection"]
N1["7.1 Distributions and conditional probability"]
N2["7.2 Intervals"]
N3["7.3 Bias and confounding"]
N4["7.5 Significance and its misuse"]
N5["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX013"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["7.1 Distributions and conditional probability"]
N2["7.2 Intervals"]
N3["7.3 Bias and confounding"]
N4["7.5 Significance and its misuse"]
N5["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX013"]
[Learning outcomes and connection]
|
v
[7.1 Distributions and conditional probability]
|
v
[7.2 Intervals]
|
v
[7.3 Bias and confounding]
|
v
[7.5 Significance and its misuse]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX013]
Learning outcomes and connection
After this chapter you can (1) compute detection measures from outcomes and attach confidence intervals; (2) apply Bayes' rule to see how base rates change the meaning of an alert; (3) recognise confounding by case mix, including Simpson's paradox; and (4) report a comparison so that its uncertainty is visible. It prepares Chapters 29 (measurement), 41 (decisions) and 42 (applied research).
7.1 Distributions and conditional probability
An alert is a test. Its sensitivity is $P(\text{alert} \mid \text{incident})$; its false-positive rate is $P(\text{alert} \mid \text{no incident})$. What a responder needs is the positive predictive value, $P(\text{incident} \mid \text{alert})$, which by Bayes' rule is
$$\text{PPV} = \frac{s\,\pi}{s\,\pi + f(1-\pi)}$$
with prevalence $\pi$. Axelsson (2000) showed that this base-rate effect dominates intrusion detection: even excellent detectors produce mostly false alarms when real attacks are rare. Alahmadi, Axon and Martinovic (2022) document how analysts experience the result.
7.2 Intervals
A proportion from a small sample is uncertain. The Wilson score interval behaves better than the simple $\hat p \pm 1.96\sqrt{\hat p(1-\hat p)/n}$ near 0 or 1 and for small $n$. Always report $k/n$, the estimate and the interval. "90% correct" from 20 cases has an interval of roughly 70–97%.
In Lab 7, workflow B's low-severity score of 5/10 has a Wilson interval from 0.237 to 0.763, so it is consistent with almost any underlying rate. Workflow A's high-severity 19/20 still allows a true rate as low as 0.764.
7.3 Bias and confounding
Comparisons between workflows are confounded when the groups differ in case mix. Simpson's paradox (Simpson, 1951) is the extreme case: a workflow can be better in every severity stratum and worse overall because it handled a harder mix. Stratify by known confounders, or better, assign cases to workflows at random when you can (Chapter 29).
The HVSS comparison in Lab 7 has exactly this structure. Workflow B's team received 70 of its 80 cases at high severity, while A's team received only 20 of 100, so pooling rewards the easier mix. Randomly assigning incoming cases to A or B would have removed that imbalance before any review panel scored the outcomes.
7.4 Worked example (synthetic): two triage workflows
HVSS ran workflow A in one team and workflow B in another. A outcome is "correct escalation decision" as judged later by a review panel.
"""Lab 7 (synthetic): detection measures with Wilson intervals, and a confounded comparison."""
import math
def wilson(k, n, z=1.96):
if n == 0: return (0.0, 1.0)
p = k / n; d = 1 + z*z/n
c = (p + z*z/(2*n)) / d; h = z*math.sqrt(p*(1-p)/n + z*z/(4*n*n)) / d
return (c - h, c + h)
# Workflow A and B outcomes, split by severity (TP = correct escalation of a true incident)
data = {"A": {"high": (19, 20), "low": (45, 80)}, "B": {"high": (60, 70), "low": (5, 10)}}
for w, d in data.items():
k = sum(v[0] for v in d.values()); n = sum(v[1] for v in d.values())
lo, hi = wilson(k, n)
print(f"{w}: pooled correct {k}/{n} = {k/n:.3f} 95% CI [{lo:.3f}, {hi:.3f}]")
for s, (k2, n2) in d.items():
lo2, hi2 = wilson(k2, n2)
print(f" {s:4s} {k2}/{n2} = {k2/n2:.3f} CI [{lo2:.3f}, {hi2:.3f}]")
# Base-rate: alert with sensitivity 0.95, false positive rate 0.02, prevalence 0.005
se, fpr, prev = 0.95, 0.02, 0.005
ppv = se*prev / (se*prev + fpr*(1-prev))
print(f"PPV at prevalence {prev}: {ppv:.3f}")
Expected output:
A: pooled correct 64/100 = 0.640 95% CI [0.542, 0.727]
high 19/20 = 0.950 CI [0.764, 0.991]
low 45/80 = 0.562 CI [0.453, 0.666]
B: pooled correct 65/80 = 0.812 95% CI [0.713, 0.883]
high 60/70 = 0.857 CI [0.757, 0.921]
low 5/10 = 0.500 CI [0.237, 0.763]
PPV at prevalence 0.005: 0.193
Pooled, B looks far better (0.81 vs 0.64). Stratified, A is better in both severities (0.95 vs 0.86 high; 0.56 vs 0.50 low). The pooled result reflects that B's team received mostly high-severity cases, which are easier to judge correctly here. Note also the width of the intervals: the low-severity comparison for B rests on 10 cases and supports no conclusion. The PPV line shows the base-rate effect: with 95% sensitivity and a 2% false-positive rate, only about one alert in five is real at 0.5% prevalence.
7.5 Significance and its misuse
A $p$-value answers a narrow question about data under a null model; it does not measure effect size or importance. Searching many metrics until one is "significant" produces false findings (Simmons, Nelson and Simonsohn, 2011). Pre-state the primary measure before looking at results.
For the HVSS triage comparison, the primary measure was correct escalation as judged by the review panel, and it should be fixed before results are opened. If the team then also tested time to first action, number of handovers and reopen rate, one of them would eventually look "significant" by chance.
7.6 Errors and limits
- Reporting accuracy without base rate or interval.
- Pooling across strata with different mixes.
- Treating the review panel's judgment as ground truth without measuring agreement.
All three errors can be seen in the Lab 7 comparison of workflows A and B. Quoting B's pooled 0.81 without its interval and case mix hides that A is better in both strata. Quoting 95% sensitivity without the 0.5% prevalence hides that only about one alert in five is real. Every 'correct' label also comes from one review panel, so a second independent reviewer on a sample is needed before agreement can be stated.
Exercises — chapter 07
EX013 — PPV. Objective: apply Bayes' rule. Inputs: sensitivity 0.90, false-positive rate 0.01; prevalence 0.001 and 0.05. Steps: compute PPV for both; state the operational meaning. Expected evidence: two values and a sentence. Rubric: 3 = 0.083 and 0.826 with interpretation; 2 = one error; 1 = formula; 0 = none.
EX014 — Fair comparison. Objective: design out confounding. Inputs: Lab 7. Steps: compute a severity-standardised accuracy for A and B using a common mix of 50% high, 50% low; propose a data-collection design that avoids the problem next time. Expected evidence: two numbers and a design. Rubric: 3 = A 0.756, B 0.679 with randomised or stratified assignment design; 2 = numbers only; 1 = attempt; 0 = none.
Chapter synthesis
Detection and response measures are proportions with uncertainty, conditioned on base rates and confounded by case mix. Report intervals, stratify or randomise, and let PPV — not sensitivity — describe what an alert means to a responder.
Continue studying — chapter 07 of 50
← Chapter 06 · Course chapter index · Chapter 08 →
Continue to the next authored lesson.