Chapter 7 — Probability and statistics
Lesson map
Lesson 07 published. Map, example, EX019.
flowchart TD
N0["Learning outcomes"]
N1["7.1 Conditional probability and Bayes"]
N2["7.2 The base-rate effect"]
N3["7.3 Estimating rates"]
N4["7.4 Distributions and independence"]
N5["7.5 Comparing two systems"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX019"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["7.1 Conditional probability and Bayes"]
N2["7.2 The base-rate effect"]
N3["7.3 Estimating rates"]
N4["7.4 Distributions and independence"]
N5["7.5 Comparing two systems"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX019"]
[Learning outcomes]
|
v
[7.1 Conditional probability and Bayes]
|
v
[7.2 The base-rate effect]
|
v
[7.3 Estimating rates]
|
v
[7.4 Distributions and independence]
|
v
[7.5 Comparing two systems]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX019]
Learning outcomes
You will (1) apply Bayes' rule to detection problems; (2) explain base rates and why rare events dominate alert precision; (3) estimate rates with confidence intervals; and (4) design evaluations with enough samples.
7.1 Conditional probability and Bayes
P(A | B) = P(B | A) P(A) / P(B). For a detector of malicious prompts, the true-positive rate is P(alert | malicious) and the false-positive rate is P(alert | benign). What operators care about is precision, P(malicious | alert), which depends strongly on the base rate P(malicious).
The two rates describe the detector and stay roughly fixed; precision describes the detector in a particular stream of traffic and changes whenever the mix changes. Sam writes all three numbers in every detector report, because operators who read only the true-positive rate tend to treat every alert as an attack.
7.2 The base-rate effect
The lab computes precision for three detectors and three prevalences. A detector with 95% TPR and 5% FPR looks good, but if only 1 in 1,000 prompts is malicious, fewer than 2% of its alerts are real. Arp et al. (arXiv:2010.09470) list the base-rate fallacy among the common pitfalls of machine learning in security.
"""Lab B: base rates and alarm precision (Bayes' rule).
A detector for malicious prompts with given true- and false-positive rates
is applied to traffic with different shares of malicious prompts."""
for tpr, fpr in ((0.95, 0.05), (0.95, 0.01), (0.80, 0.001)):
for prev in (0.10, 0.01, 0.001):
tp = tpr * prev; fp = fpr * (1 - prev)
prec = tp / (tp + fp)
per_10k = round(10000 * (tp + fp))
print(f"tpr={tpr:.2f} fpr={fpr:.3f} prevalence={prev:.3f} precision={prec:.3f} alerts_per_10k={per_10k}")
Expected output:
tpr=0.95 fpr=0.050 prevalence=0.100 precision=0.679 alerts_per_10k=1400
tpr=0.95 fpr=0.050 prevalence=0.010 precision=0.161 alerts_per_10k=590
tpr=0.95 fpr=0.050 prevalence=0.001 precision=0.019 alerts_per_10k=509
tpr=0.95 fpr=0.010 prevalence=0.100 precision=0.913 alerts_per_10k=1040
tpr=0.95 fpr=0.010 prevalence=0.010 precision=0.490 alerts_per_10k=194
tpr=0.95 fpr=0.010 prevalence=0.001 precision=0.087 alerts_per_10k=109
tpr=0.80 fpr=0.001 prevalence=0.100 precision=0.989 alerts_per_10k=809
tpr=0.80 fpr=0.001 prevalence=0.010 precision=0.890 alerts_per_10k=90
tpr=0.80 fpr=0.001 prevalence=0.001 precision=0.445 alerts_per_10k=18
Reading the table row by row shows that cutting false positives does more than raising detection. At a prevalence of 0.001, the third detector catches fewer attacks than the first, yet nearly half its alerts are real against under 2%, and it raises 18 alerts per 10,000 prompts instead of 509.
7.3 Estimating rates
An observed rate k/n has uncertainty. The Wilson score interval is reliable even for small n or rates near 0 or 1. Report intervals, and size studies so intervals are narrow enough for the decision: n ≈ z² p(1 − p)/h² for half-width h.
Wilson's interval stays inside 0 and 1 and is close to correct even when few failures are seen, which matters because good systems produce few. For a rate around 5% and a half-width of 1 percentage point at 95% confidence, the formula gives about 1.96² × 0.05 × 0.95 / 0.01², roughly 1,800 test items. Sam runs this sum before collecting data, since a study too small to settle the question wastes everyone's time.
7.4 Distributions and independence
Binomial models count successes in independent trials. Many security evaluations violate independence: prompts generated from one template are correlated, so the effective sample size is smaller. Group by template and resample groups (a cluster bootstrap) when estimating intervals.
Suppose 1,000 test prompts come from 20 templates with 50 wordings each. If a filter fails on one template, it probably fails on most of its wordings, so the evaluation has nearer 20 independent units than 1,000. Treating them as 1,000 makes the interval far too narrow. Resampling whole templates gives an honest width and also shows which templates drive the failures.
7.5 Comparing two systems
To decide whether a new filter is better than the old one, run both on the same items and analyse the paired differences; item difficulty then cancels. A paired bootstrap — resample items, recompute the difference, take percentiles — gives an interval for the improvement. If the interval includes zero, the data do not show an improvement, however attractive the point estimate.
Pairing removes the noise of item difficulty: an unusually hard prompt counts against both filters, so it no longer obscures the difference between them.
7.6 Worked example (synthetic NRS case)
Wrench's injection detector had 95% TPR and 1% FPR in testing. Sam estimated that perhaps 1 in 1,000 supplier bulletins would ever contain an injection attempt. From the lab, precision would be about 8.7% with 109 alerts per 10,000 bulletins — mostly false alarms. Sam recommended routing alerts to a lightweight review queue rather than blocking, and pairing detection with privilege limits so that missed attacks have bounded impact.
The 1-in-1,000 figure is itself an estimate, so Sam also reads the lab's 1-in-100 row, where precision rises to 0.490. Either way, blocking every flagged bulletin would mostly block legitimate ones. A review queue keeps technicians informed, and the privilege limits mean the decision does not depend on the detector being right.
7.7 Errors and limits
- Quoting accuracy instead of precision at realistic base rates.
- Treating correlated samples as independent.
- Point estimates without intervals.
Accuracy is misleading at low base rates: a detector that never alerts is 99.9% accurate when one prompt in a thousand is malicious. Correlated samples inflate confidence, as when many prompts share a template. Point estimates without intervals hide how little a small test can show. Sam's review checklist asks three questions of any metric: at what base rate, counted over which units, and with what interval.
Exercises
- EX019 — Precision. With TPR 0.90, FPR 0.02 and prevalence 0.005, compute precision. Rubric: 3 = 0.0045/(0.0045 + 0.0199) ≈ 0.184; 2 = slip; 1 = method; 0 = none.
- EX020 — Lab reading. Which detector in the lab gives precision above 0.4 at prevalence 0.001, and at what cost? Rubric: 3 = TPR 0.80/FPR 0.001: 0.445, lower TPR; 2 = row only; 1 = vague; 0 = wrong.
- EX021 — Clustering. Why does generating 500 prompts from 5 templates overstate the sample size? Rubric: 3 = correlated within template; effective n closer to 5 groups; use cluster bootstrap; 2 = partial; 1 = vague; 0 = wrong.
Synthesis
Probability tells engineers what an alert really means. Base rates, honest intervals and correct sampling units decide whether a detector is useful in operation.
Continue studying — chapter 07 of 50
← Chapter 06 (draft) · Course chapter index · Chapter 08 (draft) →