Chapter 12 — Probability and statistics
Lesson map
Lesson 12 published. Map, example, EX034.
flowchart TD
N0["Learning outcomes"]
N1["12.1 Conditional probability"]
N2["12.2 Base rates"]
N3["12.3 Confidence intervals"]
N4["12.4 Comparing systems"]
N5["12.4b Variation and sample size"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX034"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["12.1 Conditional probability"]
N2["12.2 Base rates"]
N3["12.3 Confidence intervals"]
N4["12.4 Comparing systems"]
N5["12.4b Variation and sample size"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX034"]
[Learning outcomes]
|
v
[12.1 Conditional probability]
|
v
[12.2 Base rates]
|
v
[12.3 Confidence intervals]
|
v
[12.4 Comparing systems]
|
v
[12.4b Variation and sample size]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX034]
Learning outcomes
You will (1) use conditional probability and Bayes' rule; (2) explain base rates and alert precision; (3) compute and interpret confidence intervals for rates; and (4) compare two systems fairly.
12.1 Conditional probability
P(A | B) is the probability of A given B. A detector's true-positive rate is P(alert | attack); what operators need is P(attack | alert), which Bayes' rule gives from the true- and false-positive rates and the base rate of attacks.
The two are easily confused, and vendors usually quote the first. A detector that catches 95% of attacks still produces mostly false alarms if attacks are rare, because the clean majority generates many false positives. Bayes' rule makes the calculation explicit, and Lab B shows how strongly the answer depends on the base rate.
12.2 Base rates
When attacks are rare, even good detectors produce mostly false alarms. The lab shows a detector with 95% TPR and 5% FPR giving under 2% precision at a base rate of 1 in 1,000.
"""Lab B: base rates and alarm precision (Bayes' rule).
A detector for malicious prompts with given true- and false-positive rates
is applied to traffic with different shares of malicious prompts."""
for tpr, fpr in ((0.95, 0.05), (0.95, 0.01), (0.80, 0.001)):
for prev in (0.10, 0.01, 0.001):
tp = tpr * prev; fp = fpr * (1 - prev)
prec = tp / (tp + fp)
per_10k = round(10000 * (tp + fp))
print(f"tpr={tpr:.2f} fpr={fpr:.3f} prevalence={prev:.3f} precision={prec:.3f} alerts_per_10k={per_10k}")
Expected output:
tpr=0.95 fpr=0.050 prevalence=0.100 precision=0.679 alerts_per_10k=1400
tpr=0.95 fpr=0.050 prevalence=0.010 precision=0.161 alerts_per_10k=590
tpr=0.95 fpr=0.050 prevalence=0.001 precision=0.019 alerts_per_10k=509
tpr=0.95 fpr=0.010 prevalence=0.100 precision=0.913 alerts_per_10k=1040
tpr=0.95 fpr=0.010 prevalence=0.010 precision=0.490 alerts_per_10k=194
tpr=0.95 fpr=0.010 prevalence=0.001 precision=0.087 alerts_per_10k=109
tpr=0.80 fpr=0.001 prevalence=0.100 precision=0.989 alerts_per_10k=809
tpr=0.80 fpr=0.001 prevalence=0.010 precision=0.890 alerts_per_10k=90
tpr=0.80 fpr=0.001 prevalence=0.001 precision=0.445 alerts_per_10k=18
Reading Lab B across rows: at a base rate of one in a thousand, lowering the false-positive rate from 5% to 1% raises precision from 0.019 to 0.087, and the 0.001 detector reaches 0.445. Alerts per 10,000 conversations show the workload: 509 for the first detector against 18 for the last. Reviewers can handle eighteen alerts; five hundred would be ignored.
12.3 Confidence intervals
An observed rate from a sample is uncertain. The Wilson interval works well for small samples and extreme rates. Three successes out of twenty is consistent with true rates from about 5% to 36%.
"""Lab S: reporting attack-success rates honestly: Wilson intervals,
seed variation and a paired bootstrap. Synthetic trial outcomes only."""
import math, random
def wilson(k, n, z=1.96):
p = k/n; d = 1 + z*z/n
c = (p + z*z/(2*n))/d; h = z*math.sqrt(p*(1-p)/n + z*z/(4*n*n))/d
return c - h, c + h
for k, n in ((3, 20), (30, 200), (300, 2000)):
lo, hi = wilson(k, n); print(f"successes={k}/{n} rate={k/n:.3f} wilson95=[{lo:.3f}, {hi:.3f}]")
# five seeds of a synthetic defence evaluation: per-seed attack success rate
rates = []
for seed in range(5):
r = random.Random(seed); rates.append(sum(r.random() < 0.12 for _ in range(200))/200)
m = sum(rates)/5; sd = math.sqrt(sum((x-m)**2 for x in rates)/4)
print("per-seed rates:", " ".join(f"{x:.3f}" for x in rates), f"mean={m:.3f} sd={sd:.3f}")
# paired bootstrap: baseline vs defence on the same 300 synthetic prompts
r = random.Random(42)
base = [r.random() < 0.30 for _ in range(300)]
defn = [b and r.random() < 0.6 for b in base] # defence blocks ~40% of successes
diffs = []
for _ in range(2000):
idx = [r.randrange(300) for _ in range(300)]
diffs.append(sum(base[i] for i in idx)/300 - sum(defn[i] for i in idx)/300)
diffs.sort()
print(f"baseline_asr={sum(base)/300:.3f} defended_asr={sum(defn)/300:.3f} "
f"reduction_ci95=[{diffs[50]:.3f}, {diffs[1949]:.3f}]")
Expected output:
successes=3/20 rate=0.150 wilson95=[0.052, 0.360]
successes=30/200 rate=0.150 wilson95=[0.107, 0.206]
successes=300/2000 rate=0.150 wilson95=[0.135, 0.166]
per-seed rates: 0.110 0.130 0.135 0.105 0.145 mean=0.125 sd=0.017
baseline_asr=0.350 defended_asr=0.190 reduction_ci95=[0.120, 0.203]
The lab shows how the interval narrows with sample size: the same 15% rate gives an interval from about 11% to 21% with 200 trials and from 0.135 to 0.166 with 2,000. A demonstration with 20 prompts cannot distinguish a weak defence from a strong one. The per-seed line adds another source of spread: five runs of the same evaluation range from 0.105 to 0.145.
12.4 Comparing systems
To compare two defences, test both on the same items and look at the paired difference with a bootstrap interval. If the interval includes zero, you cannot claim one is better.
Testing on the same items removes the variation caused by some prompts being harder than others. In Lab S, the defence lowers attack success from 0.350 to 0.190, and the bootstrap interval for the reduction runs from 0.120 to 0.203. Because it excludes zero, the improvement is real for this test set; whether it holds on other prompts is a separate question.
12.4b Variation and sample size
Results also vary with random seeds, data splits and prompt wording. Run several seeds and report their spread, as the lab's per-seed line shows. To plan how many test cases you need for a rate near p with interval half-width h at 95% confidence, use n ≈ 1.96² × p(1 − p) / h². For p = 0.1 and h = 0.02 that is about 864 cases — far more than most quick demonstrations use, which is why many published "success rates" are less certain than they look.
The formula also shows the cost of precision: halving h to 0.01 needs four times as many cases, about 3,457.
12.5 Worked example (synthetic LCH case)
A vendor offered a "prompt-attack detector" with 98% accuracy. Rina asked for TPR and FPR: 90% and 2%. With an estimated 1 malicious conversation per 2,000, precision would be about 2% — most alerts false. LCH used the detector only to flag conversations for weekly review, not to block patients.
The 98% accuracy figure was almost meaningless here, since a detector that never fired would score about 99.95% on the same traffic. Using it for weekly review rather than blocking means false alerts cost reviewer time, not patient access.
12.6 Errors and limits
- Accuracy without base rates.
- Single numbers without intervals.
- Comparing systems on different test sets.
Accuracy hides the trade-off between missed attacks and false alarms, and can look high for a useless detector when attacks are rare. A single number without an interval invites overconfidence, especially from small tests such as three successes in twenty. Comparing systems on different test sets confuses differences in the systems with differences in the prompts. Lab S shows the fix for each, using the same synthetic prompts.
Exercises
- EX034 — Precision. Compute precision for TPR 0.90, FPR 0.02, base rate 0.0005. Rubric: 3 = 0.00045/(0.00045 + 0.01999) ≈ 0.022; 2 = slip; 1 = method; 0 = none.
- EX035 — Lab reading. What is the Wilson interval for 300/2000 in Lab S? Rubric: 3 = [0.135, 0.166]; 2 = approximate; 1 = point only; 0 = wrong.
- EX036 — Paired comparison. Why test two defences on the same items? Rubric: 3 = removes item-difficulty variation; valid paired inference; 2 = partial; 1 = vague; 0 = wrong.
Synthesis
Probability tells you what an alert means and how sure you can be. Base rates, intervals and paired comparisons are the minimum for honest AI-security claims.
Continue studying — chapter 12 of 40
← Chapter 11 (draft) · Course chapter index · Chapter 13 (draft) →