Chapter 23 — AI and machine-learning security: evaluation
Lesson map
Lesson 23 published. Map, example, EX067.
flowchart TD
N0["Learning outcomes"]
N1["23.1 Planning"]
N2["23.2 Adaptive testing"]
N3["23.3 Reporting"]
N4["23.4 Residual risk"]
N5["23.4b Presenting to owners"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX067"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["23.1 Planning"]
N2["23.2 Adaptive testing"]
N3["23.3 Reporting"]
N4["23.4 Residual risk"]
N5["23.4b Presenting to owners"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX067"]
[Learning outcomes]
|
v
[23.1 Planning]
|
v
[23.2 Adaptive testing]
|
v
[23.3 Reporting]
|
v
[23.4 Residual risk]
|
v
[23.4b Presenting to owners]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX067]
Learning outcomes
You will (1) plan a full security evaluation of an AI system; (2) include adaptive testing; (3) report results with intervals and side effects; and (4) state residual risk.
23.1 Planning
Start from the risk scenarios. For each, define the attacker goal, the measurement (effect, not words), the suites (development, held-out, adaptive), sample sizes and pass criteria. Fix versions. Get authorisation for the environment. Write the plan down before running anything, so criteria cannot drift to fit results. Jonas's quarterly Ask Brightwell evaluation follows this order: each scenario gets a measure from tool logs, a held-out and an adaptive suite, and pass criteria written before the run. The written authorisation covers staging only, which the plan states.
23.2 Adaptive testing
Fixed suites measure known attacks. Adaptive testing gives testers knowledge of the defences and time to work around them. Carlini et al. (arXiv:1902.06705) and Athalye et al. (arXiv:1802.00420) showed that defences which looked strong on fixed attacks failed against adapted ones. Ask colleagues who did not build the defences to try. At Brightwell, adaptive cases come from testers who did not build the policy service. The adaptive case that started a return on the last valid day with a different item found a gap fixed suites had not.
23.3 Reporting
For each scenario: successes/trials, rate, 95% interval, comparison with the previous version, side effects on normal use (false refusals, completion), conditions (versions, dates). Include failed cases in an appendix so developers can reproduce them. Jonas's quarterly report on Ask Brightwell gives harmful returns as counts out of 80 held-out and 60 adaptive cases, canary leaks out of 50, and normal completion out of 500, each with versions. The failed adaptive case is in the appendix with steps. Each failed case in the appendix lists the versions it ran against. Every table shows each count with its interval, so a reader can see how few adaptive cases lie behind a rate.
23.4 Residual risk
State what remains: scenarios not tested, attacks that succeeded, limits of the suites (languages, modalities), and dependencies on vendor behaviour. Recommend owners and next steps. For Ask Brightwell, the quarterly evaluation names non-English injections as untested and vendor model updates as a dependency. Each gets an owner: Jonas's team to build non-English cases, and the platform team to watch vendor notices and rerun the gate after any update. Residual risk without an owner is a list nobody acts on. Owners decide whether to fund, accept or defer each item.
23.4b Presenting to owners
Owners need a one-page summary: which scenarios pass, which fail, what changed since the last evaluation, and which decisions they must take. Put details in appendices. Avoid a single overall "score", because it hides which risks remain. For Ask Brightwell, the one-page summary told the product owner that refund fraud passed held-out tests, one adaptive case had been fixed, and non-English injections were untested, with a decision on whether to fund those cases. The summary fits on one page so that owners read it before the meeting, and the appendices stay available for anyone who wants detail.
23.5 Worked example (synthetic Brightwell case)
Jonas's quarterly evaluation of Ask Brightwell: harmful returns 0/80 held-out and 1/60 adaptive (the adaptive case abused a policy edge: a return started on the last valid day with a different item — fixed by matching item IDs); canary leaks 0/50; cost floods held within limits; normal completion 96% of 500. Residual risks: non-English injections untested; vendor model updates may change behaviour. The adaptive result shows why fixed suites are not enough, and the residual-risk list tells owners where the evaluation stops. Each residual risk has an owner and a next step.
23.6 Errors and limits
- Only fixed suites.
- No side-effect measurement.
- Missing residual-risk section.
Jonas's evaluation avoids these by combining held-out and adaptive cases, measuring normal completion beside attack rates, and listing residual risks in a separate part of the report. A report without side effects would hide a defence that refuses too much. A report without residual risks lets owners believe everything was tested. Brightwell's report template includes a fixed heading for residual risk, so the author must write something there even when the answer is that nothing remained untested.
Exercises
- EX067 — Plan. Write an evaluation plan table for three Ask Brightwell scenarios. Rubric: 3 = goal, measurement, suites, size, criteria for each; 2 = two scenarios; 1 = one; 0 = none.
- EX068 — Adaptive. Why did the adaptive case find a flaw the held-out suite missed? Rubric: 3 = tester knew the rules, targeted edge; 2 = partial; 1 = vague; 0 = none.
- EX069 — Residual. Write two residual-risk statements. Rubric: 3 = two with owner and next step; 2 = without owners; 1 = one; 0 = none.
Synthesis
A full evaluation is planned from scenarios, includes adaptive testers, reports effects with intervals and side effects, and ends with honest residual risk.
Continue studying — chapter 23 of 30
← Chapter 22 (draft) · Course chapter index · Chapter 24 (draft) →