Chapter 29 — AI and machine-learning security: measurement
Lesson map
Lesson 29 published. Map, example, EX085.
flowchart TD
N0["Learning outcomes"]
N1["29.1 Metrics that match properties"]
N2["29.2 Uncertainty"]
N3["29.3 Pitfalls"]
N4["29.4 Measurement programmes"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX085"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["29.1 Metrics that match properties"]
N2["29.2 Uncertainty"]
N3["29.3 Pitfalls"]
N4["29.4 Measurement programmes"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX085"]
[Learning outcomes]
|
v
[29.1 Metrics that match properties]
|
v
[29.2 Uncertainty]
|
v
[29.3 Pitfalls]
|
v
[29.4 Measurement programmes]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX085]
Learning outcomes
You will (1) choose metrics for AI-security properties (attack success rate, robust accuracy, TPR at fixed FPR, utility cost); (2) estimate them with intervals and paired comparisons; (3) avoid measurement pitfalls (leakage, weak attacks, base rates, LLM-judge bias); and (4) design measurement programmes that support release decisions.
29.1 Metrics that match properties
- Integrity under attack: attack success rate (ASR) or robust accuracy at a stated budget.
- Detection: TPR at a fixed, operationally acceptable FPR; precision at realistic base rates.
- Privacy: membership-inference TPR at low FPR, AUC as a secondary summary.
- Cost: clean accuracy, task success, latency, false refusals.
Always pair a security metric with a cost metric measured on the same items.
Pairing prevents an easy win: any system can reach zero attack success by refusing every request, and only a cost metric on the same items reveals it. Measuring both on one set also lets the two be read together, item by item.
29.2 Uncertainty
Use Wilson intervals for rates, bootstrap for averaged metrics, and paired bootstrap for comparisons. Report seeds and their spread.
"""Lab S: reporting attack-success rates honestly: Wilson intervals,
seed variation and a paired bootstrap. Synthetic trial outcomes only."""
import math, random
def wilson(k, n, z=1.96):
p = k/n; d = 1 + z*z/n
c = (p + z*z/(2*n))/d; h = z*math.sqrt(p*(1-p)/n + z*z/(4*n*n))/d
return c - h, c + h
for k, n in ((3, 20), (30, 200), (300, 2000)):
lo, hi = wilson(k, n); print(f"successes={k}/{n} rate={k/n:.3f} wilson95=[{lo:.3f}, {hi:.3f}]")
# five seeds of a synthetic defence evaluation: per-seed attack success rate
rates = []
for seed in range(5):
r = random.Random(seed); rates.append(sum(r.random() < 0.12 for _ in range(200))/200)
m = sum(rates)/5; sd = math.sqrt(sum((x-m)**2 for x in rates)/4)
print("per-seed rates:", " ".join(f"{x:.3f}" for x in rates), f"mean={m:.3f} sd={sd:.3f}")
# paired bootstrap: baseline vs defence on the same 300 synthetic prompts
r = random.Random(42)
base = [r.random() < 0.30 for _ in range(300)]
defn = [b and r.random() < 0.6 for b in base] # defence blocks ~40% of successes
diffs = []
for _ in range(2000):
idx = [r.randrange(300) for _ in range(300)]
diffs.append(sum(base[i] for i in idx)/300 - sum(defn[i] for i in idx)/300)
diffs.sort()
print(f"baseline_asr={sum(base)/300:.3f} defended_asr={sum(defn)/300:.3f} "
f"reduction_ci95=[{diffs[50]:.3f}, {diffs[1949]:.3f}]")
Expected output:
successes=3/20 rate=0.150 wilson95=[0.052, 0.360]
successes=30/200 rate=0.150 wilson95=[0.107, 0.206]
successes=300/2000 rate=0.150 wilson95=[0.135, 0.166]
per-seed rates: 0.110 0.130 0.135 0.105 0.145 mean=0.125 sd=0.017
baseline_asr=0.350 defended_asr=0.190 reduction_ci95=[0.120, 0.203]
The lab's first three lines show the same 15% rate with intervals narrowing as trials grow: from 5–36% at 20 trials to 13.5–16.6% at 2,000. The seed spread of 0.017 shows how much a single run can move. The paired bootstrap gives a reduction interval of 0.120 to 0.203, which excludes zero, so the defence's improvement is supported by the data rather than by one lucky sample.
29.3 Pitfalls
Leakage between tuning and test sets inflates results; weak or non-adaptive attacks make defences look strong (Athalye et al., arXiv:1802.00420); ignoring base rates misleads about detector usefulness (Arp et al., arXiv:2010.09470). When an LLM is used as a judge of outputs, validate it against human labels on a sample and report agreement; judges can be biased toward fluent or refusing answers.
Each pitfall makes results look better than they are, which is why reviewers look for them first. A surprisingly good number deserves the most scrutiny.
29.4 Measurement programmes
A release decision needs a fixed suite (held out, versioned), thresholds tied to requirements, a schedule (each release and periodically), and ownership. Track metrics over time to detect regressions and drift. Store raw outputs so that metrics can be recomputed if definitions change.
Ownership prevents the suite from decaying: someone must add new attack patterns, retire broken cases and keep thresholds aligned with requirements. Versioning means a change to the suite is visible, and results from different versions are not compared as if they were the same test. Raw outputs let analysts answer new questions later, such as how often a model refused benign tasks, without rerunning the whole campaign.
29.5 Worked example (synthetic NRS case)
Sam's measurement plan for Wrench: injection ASR on 600 held-out tasks (requirement ≤ 2% proposals, 0 executions), false-confirmation rate on 600 benign tasks (≤ 5%), task success on 300 planner tasks (≥ 90%), each with Wilson intervals; LLM-judge labels validated on 150 human-labelled outputs (agreement 93%). Results feed the CI gate and a quarterly report.
The plan pairs each security metric with a cost metric: injection success against false confirmations, which measures how often benign work is slowed. Requiring zero executions alongside the proposal limit reflects the architecture, since proposals can be stopped by the enforcement point while executions must never happen. A 93% agreement on 150 outputs shows the judge is usable but imperfect, so disagreements are reviewed by a person each quarter.
29.6 Errors and limits
- Security metrics without cost metrics.
- Unvalidated LLM judges.
- Changing the suite between versions and comparing numbers anyway.
A security metric alone can be improved by making the system refuse everything. An LLM judge that has not been checked against human labels may reward confident or refusing answers. Comparing numbers across different suites mixes changes in the system with changes in the test. Sam's quarterly report states the suite version on every chart and starts a new series whenever the suite changes.
Exercises
- EX085 — Metric choice. Choose primary and cost metrics for a deepfake-voice detector at a call centre. Rubric: 3 = TPR at fixed low FPR/precision at base rate plus false-reject rate/latency; 2 = one; 1 = accuracy only; 0 = none.
- EX086 — Lab reading. What is the Wilson interval for 30/200, and what does it allow you to claim? Rubric: 3 = [0.107, 0.206]; rate plausibly 11–21%; 2 = interval only; 1 = vague; 0 = wrong.
- EX087 — Judge validation. Describe how to validate an LLM judge. Rubric: 3 = human-label a random sample, compute agreement/confusion, check bias by condition, report; 2 = partial; 1 = vague; 0 = none.
Synthesis
Measurement turns AI security into decisions: matched metrics with costs, honest intervals, adaptive attacks, validated judges and a stable, versioned programme.
Continue studying — chapter 29 of 50
← Chapter 28 (draft) · Course chapter index · Chapter 30 (draft) →