Chapter 40 — Verification laboratory II
Lesson map
Lesson 40 published. Map, example, EX118.
flowchart TD
N0["Learning outcomes"]
N1["40.1 Integrated campaigns"]
N2["40.2 Acceptance criteria"]
N3["40.3 Base rates in acceptance"]
N4["40.4 Evidence package"]
N5["40.6b Repeating the campaign after fixes"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX118"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["40.1 Integrated campaigns"]
N2["40.2 Acceptance criteria"]
N3["40.3 Base rates in acceptance"]
N4["40.4 Evidence package"]
N5["40.6b Repeating the campaign after fixes"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX118"]
[Learning outcomes]
|
v
[40.1 Integrated campaigns]
|
v
[40.2 Acceptance criteria]
|
v
[40.3 Base rates in acceptance]
|
v
[40.4 Evidence package]
|
v
[40.6b Repeating the campaign after fixes]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX118]
Learning outcomes
You will (1) run an integrated verification campaign across several AI components; (2) test controls under combined attacks and failures; (3) use statistical acceptance criteria; and (4) produce an evidence package suitable for an independent assessor.
40.1 Integrated campaigns
Lab I tested components; Lab II tests the system. Combine attacks with failures: injection during a connector outage; poisoning detected during a model update; extraction attempts while monitoring is degraded. Systems often fail at these intersections.
Component tests assume every neighbour behaves normally. In the real system, outages, updates and attacks overlap, and the handling of one can open a gap for another. A connector failing may push alerts into a queue nobody watches; a model update may briefly disable monitoring. Integrated scenarios are chosen from the threat model and the failure analysis together, pairing each high-ranked threat with a plausible concurrent failure.
40.2 Acceptance criteria
Use statistical criteria tied to requirements: for a rate requirement ≤ p with confidence 95%, the upper Wilson bound (or exact bound) must be below p. With zero failures, about 3/p trials are needed (rule of three). Decide criteria before running.
"""Lab S: reporting attack-success rates honestly: Wilson intervals,
seed variation and a paired bootstrap. Synthetic trial outcomes only."""
import math, random
def wilson(k, n, z=1.96):
p = k/n; d = 1 + z*z/n
c = (p + z*z/(2*n))/d; h = z*math.sqrt(p*(1-p)/n + z*z/(4*n*n))/d
return c - h, c + h
for k, n in ((3, 20), (30, 200), (300, 2000)):
lo, hi = wilson(k, n); print(f"successes={k}/{n} rate={k/n:.3f} wilson95=[{lo:.3f}, {hi:.3f}]")
# five seeds of a synthetic defence evaluation: per-seed attack success rate
rates = []
for seed in range(5):
r = random.Random(seed); rates.append(sum(r.random() < 0.12 for _ in range(200))/200)
m = sum(rates)/5; sd = math.sqrt(sum((x-m)**2 for x in rates)/4)
print("per-seed rates:", " ".join(f"{x:.3f}" for x in rates), f"mean={m:.3f} sd={sd:.3f}")
# paired bootstrap: baseline vs defence on the same 300 synthetic prompts
r = random.Random(42)
base = [r.random() < 0.30 for _ in range(300)]
defn = [b and r.random() < 0.6 for b in base] # defence blocks ~40% of successes
diffs = []
for _ in range(2000):
idx = [r.randrange(300) for _ in range(300)]
diffs.append(sum(base[i] for i in idx)/300 - sum(defn[i] for i in idx)/300)
diffs.sort()
print(f"baseline_asr={sum(base)/300:.3f} defended_asr={sum(defn)/300:.3f} "
f"reduction_ci95=[{diffs[50]:.3f}, {diffs[1949]:.3f}]")
Expected output:
successes=3/20 rate=0.150 wilson95=[0.052, 0.360]
successes=30/200 rate=0.150 wilson95=[0.107, 0.206]
successes=300/2000 rate=0.150 wilson95=[0.135, 0.166]
per-seed rates: 0.110 0.130 0.135 0.105 0.145 mean=0.125 sd=0.017
baseline_asr=0.350 defended_asr=0.190 reduction_ci95=[0.120, 0.203]
The rule of three makes the cost visible: showing a failure rate below 1% with no failures needs about 300 trials, and below 0.1% about 3,000. Writing the criterion first stops the team from relaxing a threshold once results are in, or from stopping a test early because the numbers happen to look good. Each criterion names its requirement, so a failed bound points directly at what must change.
40.3 Base rates in acceptance
Detector requirements must be checked at realistic base rates; a detector that passes TPR/FPR thresholds may still produce unusable precision in operation.
"""Lab B: base rates and alarm precision (Bayes' rule).
A detector for malicious prompts with given true- and false-positive rates
is applied to traffic with different shares of malicious prompts."""
for tpr, fpr in ((0.95, 0.05), (0.95, 0.01), (0.80, 0.001)):
for prev in (0.10, 0.01, 0.001):
tp = tpr * prev; fp = fpr * (1 - prev)
prec = tp / (tp + fp)
per_10k = round(10000 * (tp + fp))
print(f"tpr={tpr:.2f} fpr={fpr:.3f} prevalence={prev:.3f} precision={prec:.3f} alerts_per_10k={per_10k}")
Expected output:
tpr=0.95 fpr=0.050 prevalence=0.100 precision=0.679 alerts_per_10k=1400
tpr=0.95 fpr=0.050 prevalence=0.010 precision=0.161 alerts_per_10k=590
tpr=0.95 fpr=0.050 prevalence=0.001 precision=0.019 alerts_per_10k=509
tpr=0.95 fpr=0.010 prevalence=0.100 precision=0.913 alerts_per_10k=1040
tpr=0.95 fpr=0.010 prevalence=0.010 precision=0.490 alerts_per_10k=194
tpr=0.95 fpr=0.010 prevalence=0.001 precision=0.087 alerts_per_10k=109
tpr=0.80 fpr=0.001 prevalence=0.100 precision=0.989 alerts_per_10k=809
tpr=0.80 fpr=0.001 prevalence=0.010 precision=0.890 alerts_per_10k=90
tpr=0.80 fpr=0.001 prevalence=0.001 precision=0.445 alerts_per_10k=18
The acceptance plan therefore states an expected prevalence for each detector, taken from operating data or a justified estimate, and computes precision and alert volume at that rate. A detector that would send operators a hundred false alerts for every real one fails acceptance even if its TPR and FPR meet their thresholds, because in operation its alerts would soon be ignored.
40.4 Evidence package
Organise evidence by requirement: test plan, raw results, analysis, pass/fail, deviations, re-tests. Include environment descriptions, model and data versions with hashes, and tool versions. An independent assessor should be able to follow each requirement to its evidence without asking the team.
Hashes of models and data tie each result to exact artefacts, so nobody can later argue that a different version was tested. Deviations, such as a scenario run with fewer trials than planned, are listed rather than hidden. An assessor reading the package should be able to rerun any test from the recorded versions and seeds and reach the same numbers.
40.5 Worked example (synthetic NRS case)
Sam's integrated campaign ran 18 scenarios across TrackEye, Pulse and Wrench. Scenario 11 (Pulse sensor spoofing during a ticketing-connector outage) revealed that when ticketing failed, Pulse alerts were queued silently for 40 minutes. The fix added an alert-path health check with fallback to SMS for critical alerts. All rate requirements were evaluated with predefined Wilson upper bounds; two that had passed point estimates failed their bounds and needed larger test sets, after which one passed and one led to a model improvement.
The silent queue is the sort of failure no component test could find.
40.6 Errors and limits
- Only testing components in isolation.
- Acceptance criteria decided after seeing results.
- Evidence scattered across personal folders.
Isolated component tests miss the interactions where integrated systems fail, as scenario 11 showed. Criteria chosen after seeing results can be bent to fit whatever happened. Evidence kept in personal folders disappears when people leave and cannot be reviewed by an assessor. Sam's campaign template fixes criteria, scenarios and evidence locations before the first run, and a reviewer signs it. Changes to it are dated.
40.6b Repeating the campaign after fixes
When Laboratory II finds a failure, the fix is followed by the same campaign, not a smaller one, so NRS can show the fix worked without breaking other acceptance criteria. Both runs are kept in the evidence package with their dates and versions.
Scenario 11 shows why. The fix for Pulse's silent 40-minute alert queue added a health check and an SMS fallback, which changed the alert path for every critical alert, not only Pulse's. Rerunning all 18 scenarios, not just scenario 11, was the only way to show that TrackEye's and Wrench's alerts still arrived on time through the changed path.
Exercises
- EX118 — Combined scenario. Design one combined attack-plus-failure scenario for Wrench. Rubric: 3 = specific attack + failure + expected safe behaviour + measurement; 2 = partial; 1 = vague; 0 = none.
- EX119 — Bound. You observe 1 failure in 200 trials for a requirement ≤ 2%. Does the 95% upper bound pass? (Wilson upper ≈ 2.8%.) Rubric: 3 = no, upper bound above 2%, more trials or improvement needed; 2 = answer only; 1 = uses point estimate; 0 = none.
- EX120 — Lab reading. At prevalence 0.01, what precision does the 0.95/0.01 detector give, and is it acceptable for automatic blocking? Rubric: 3 = 0.490; likely not for blocking without review, depends on cost; 2 = number only; 1 = vague; 0 = wrong.
Synthesis
Integrated verification finds failures at the intersections, judges results with predefined statistical criteria, and packages evidence so others can assess it independently.
Continue studying — chapter 40 of 50
← Chapter 39 (draft) · Course chapter index · Chapter 41 (draft) →