Chapter 9 — Evidence and measurement
Lesson map
Lesson 09 published. Map, example, EX025.
flowchart TD
N0["Learning outcomes"]
N1["9.1 Rates, not anecdotes"]
N2["9.2 Confidence intervals"]
N3["9.3 Base rates"]
N4["9.4 Reproducible evidence"]
N5["9.4b Comparing two defences"]
N6["9.4c Measuring effects"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX025"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["9.1 Rates, not anecdotes"]
N2["9.2 Confidence intervals"]
N3["9.3 Base rates"]
N4["9.4 Reproducible evidence"]
N5["9.4b Comparing two defences"]
N6["9.4c Measuring effects"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX025"]
[Learning outcomes]
|
v
[9.1 Rates, not anecdotes]
|
v
[9.2 Confidence intervals]
|
v
[9.3 Base rates]
|
v
[9.4 Reproducible evidence]
|
v
[9.4b Comparing two defences]
|
v
[9.4c Measuring effects]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX025]
Learning outcomes
You will (1) measure security outcomes as rates over trials; (2) compute and read confidence intervals; (3) understand base rates and false alarms; and (4) keep reproducible evidence.
9.1 Rates, not anecdotes
"I tried five injections and none worked" says little. Run many trials, count successes, and report the rate with its sample size. Measure what happened (a tool call, a data leak), not what the model said. Jonas's 120 Ask Brightwell cases follow this rule: success is graded from tool logs, not from what the reply said, and the report gives counts out of the number of trials. A reply that sounds like a refusal while a return was in fact started counts as a success for the attacker.
9.2 Confidence intervals
An interval shows the plausible range of the true rate. Three successes in 20 trials gives about 5–36%; 300 in 2,000 gives about 13.5–16.6%. Zero failures in n trials still allows up to about 3/n.
"""Lab S: reporting attack-success rates honestly: Wilson intervals,
seed variation and a paired bootstrap. Synthetic trial outcomes only."""
import math, random
def wilson(k, n, z=1.96):
p = k/n; d = 1 + z*z/n
c = (p + z*z/(2*n))/d; h = z*math.sqrt(p*(1-p)/n + z*z/(4*n*n))/d
return c - h, c + h
for k, n in ((3, 20), (30, 200), (300, 2000)):
lo, hi = wilson(k, n); print(f"successes={k}/{n} rate={k/n:.3f} wilson95=[{lo:.3f}, {hi:.3f}]")
# five seeds of a synthetic defence evaluation: per-seed attack success rate
rates = []
for seed in range(5):
r = random.Random(seed); rates.append(sum(r.random() < 0.12 for _ in range(200))/200)
m = sum(rates)/5; sd = math.sqrt(sum((x-m)**2 for x in rates)/4)
print("per-seed rates:", " ".join(f"{x:.3f}" for x in rates), f"mean={m:.3f} sd={sd:.3f}")
# paired bootstrap: baseline vs defence on the same 300 synthetic prompts
r = random.Random(42)
base = [r.random() < 0.30 for _ in range(300)]
defn = [b and r.random() < 0.6 for b in base] # defence blocks ~40% of successes
diffs = []
for _ in range(2000):
idx = [r.randrange(300) for _ in range(300)]
diffs.append(sum(base[i] for i in idx)/300 - sum(defn[i] for i in idx)/300)
diffs.sort()
print(f"baseline_asr={sum(base)/300:.3f} defended_asr={sum(defn)/300:.3f} "
f"reduction_ci95=[{diffs[50]:.3f}, {diffs[1949]:.3f}]")
Expected output:
successes=3/20 rate=0.150 wilson95=[0.052, 0.360]
successes=30/200 rate=0.150 wilson95=[0.107, 0.206]
successes=300/2000 rate=0.150 wilson95=[0.135, 0.166]
per-seed rates: 0.110 0.130 0.135 0.105 0.145 mean=0.125 sd=0.017
baseline_asr=0.350 defended_asr=0.190 reduction_ci95=[0.120, 0.203]
Jonas reports Ask Brightwell results this way: after the policy service, zero harmful returns in 80 held-out cases still allows a true rate up to about 4%, so he writes the bound, not "zero risk". For Ask Brightwell, Jonas computes these with the Wilson method in his harness, so every nightly report shows an interval beside each count, including the cases with zero successes.
9.3 Base rates
A detector with 95% detection and 5% false positives sounds good, but if only 1 in 1,000 events is malicious, fewer than 2% of its alerts are real. Always ask how common the bad event is.
"""Lab B: base rates and alarm precision (Bayes' rule).
A detector for malicious prompts with given true- and false-positive rates
is applied to traffic with different shares of malicious prompts."""
for tpr, fpr in ((0.95, 0.05), (0.95, 0.01), (0.80, 0.001)):
for prev in (0.10, 0.01, 0.001):
tp = tpr * prev; fp = fpr * (1 - prev)
prec = tp / (tp + fp)
per_10k = round(10000 * (tp + fp))
print(f"tpr={tpr:.2f} fpr={fpr:.3f} prevalence={prev:.3f} precision={prec:.3f} alerts_per_10k={per_10k}")
Expected output:
tpr=0.95 fpr=0.050 prevalence=0.100 precision=0.679 alerts_per_10k=1400
tpr=0.95 fpr=0.050 prevalence=0.010 precision=0.161 alerts_per_10k=590
tpr=0.95 fpr=0.050 prevalence=0.001 precision=0.019 alerts_per_10k=509
tpr=0.95 fpr=0.010 prevalence=0.100 precision=0.913 alerts_per_10k=1040
tpr=0.95 fpr=0.010 prevalence=0.010 precision=0.490 alerts_per_10k=194
tpr=0.95 fpr=0.010 prevalence=0.001 precision=0.087 alerts_per_10k=109
tpr=0.80 fpr=0.001 prevalence=0.100 precision=0.989 alerts_per_10k=809
tpr=0.80 fpr=0.001 prevalence=0.010 precision=0.890 alerts_per_10k=90
tpr=0.80 fpr=0.001 prevalence=0.001 precision=0.445 alerts_per_10k=18
ReturnGuard shows the same effect on a smaller scale. Fraud is about 1% of returns, so even a model that catches most fraud flags far more honest customers than fraudsters at a 5% false-positive rate. Jonas's recommendation of human review before refusal follows from that arithmetic, not from distrust of the model.
9.4 Reproducible evidence
Record versions (model, prompt, code, data), seeds, test suites and dates. Store scripts with results so others can re-run them. Ask Brightwell's nightly harness report records the vendor model version, the prompt version and hash, the index build, the suite version and the date at the top of every run. When a later prompt change made harmful proposals rise, those fields showed exactly which change to undo, and anyone could rerun the comparison from the stored scripts. Without those fields, the regression would have been traced by guesswork.
9.4b Comparing two defences
To compare defences fairly, run both on the same test suite, with the same model and prompt versions, and report each rate with its interval. If the intervals overlap heavily, the difference may be noise; add more trials before claiming one is better. Also measure the cost to normal users, such as wrongly blocked requests, because a defence that blocks everything scores perfectly on attacks. Jonas applied this when a vendor offered a prompt firewall: he ran it on his held-out suite and measured both what it blocked and the normal questions it wrongly stopped.
9.4c Measuring effects
Decide in advance what counts as success for the attacker. For an agent this is usually an action — a tool call with harmful parameters — rather than words in a reply. Write the grader as code that inspects tool logs, so results do not depend on a person's judgement of text. For Ask Brightwell, the attacker's success is a start_return call for an order the signed-in user did not ask about. The grader reads the tool log for such calls; it does not look at the reply text, so a reply that sounds harmless while starting a return is still counted as a success.
9.5 Worked example (synthetic Brightwell case)
ReturnGuard flags fraudulent returns. Jonas finds fraud in about 1% of returns; the model's false-positive rate is 5%. Using the lab, precision is about 16%: five of six flagged customers are honest. He recommends a 1% false-positive threshold (precision 0.490 in the lab, so reviewers see roughly one honest customer for every fraudster flagged) and human review before any refusal. The recommendation does not claim the model is bad; it shows that a rare event makes most alerts wrong at that threshold. Human review protects honest customers from the errors that remain at the stricter threshold too.
9.6 Errors and limits
- Rates without sample sizes.
- Ignoring base rates.
- Unreproducible test runs.
Jonas's reports avoid all three: every Ask Brightwell rate is given as a count out of trials, ReturnGuard's alerts are discussed with the 1% fraud rate in view, and each run records versions and seeds so the numbers can be produced again. A figure without these is not evidence anyone can check or act on. The harness enforces all three automatically, so no manual step can skip them. Without recorded seeds, a rerun could differ and nobody could tell why.
Exercises
- EX025 — Interval. What can you claim from 0 failures in 600 trials? Rubric: 3 = upper bound ≈0.5%, assumptions; 2 = number only; 1 = "it's safe"; 0 = none.
- EX026 — Base rate. Use the lab to explain ReturnGuard's precision at 5% and 1% false positives. Rubric: 3 = 0.161 and 0.490 with meaning; 2 = numbers; 1 = one; 0 = wrong.
- EX027 — Evidence record. List what to record for a reproducible evaluation. Rubric: 3 = versions, seeds, suites, dates, scripts; 2 = four; 1 = two; 0 = none.
Synthesis
Good evidence counts outcomes over many trials, shows uncertainty, respects base rates and can be reproduced.
Continue studying — chapter 09 of 30
← Chapter 08 (draft) · Course chapter index · Chapter 10 (draft) →