◇ OpenCyber Academy
Bachelor's · Chapter 27 · English

Chapter 27 — AI and machine-learning security: verification

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 27 — AI and machine-learning security: verification

Lesson map

Lesson 27 published. Map, example, EX079.

flowchart TD
  N0["Learning outcomes"]
  N1["27.1 Requirements to verify"]
  N2["27.2 Acceptance criteria"]
  N3["27.3 End-to-end verification"]
  N4["27.4 Reports"]
  N5["27.5 Regression"]
  N6["27.5b Verifying versions, not names"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX079"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["27.1 Requirements to verify"]
  N2["27.2 Acceptance criteria"]
  N3["27.3 End-to-end verification"]
  N4["27.4 Reports"]
  N5["27.5 Regression"]
  N6["27.5b Verifying versions, not names"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX079"]

[Learning outcomes]
    |
    v
[27.1 Requirements to verify]
    |
    v
[27.2 Acceptance criteria]
    |
    v
[27.3 End-to-end verification]
    |
    v
[27.4 Reports]
    |
    v
[27.5 Regression]
    |
    v
[27.5b Verifying versions, not names]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX079]

Learning outcomes

You will (1) verify that AI-security requirements are met with statistical evidence; (2) define acceptance criteria with confidence bounds; (3) verify controls end to end; and (4) write verification reports.

27.1 Requirements to verify

Requirements such as "no booking for a patient other than the logged-in patient or carer" are verified by policy tests (deterministic) and adversarial suites (statistical). "Leakage rate below 1% with 95% confidence" requires enough cases: with zero failures, about 300 cases give an upper bound near 1% (rule of three).

The policy test is checked once per rule, because the outcome does not vary; the adversarial suite is repeated because the model's behaviour does. Both are needed, since a policy test cannot show how often the model tries a forbidden action.

27.2 Acceptance criteria

Decide before testing: metric, sample size, threshold and confidence. Use Wilson upper bounds rather than point estimates when deciding pass/fail.

"""Lab S: reporting attack-success rates honestly: Wilson intervals,
seed variation and a paired bootstrap. Synthetic trial outcomes only."""
import math, random
def wilson(k, n, z=1.96):
    p = k/n; d = 1 + z*z/n
    c = (p + z*z/(2*n))/d; h = z*math.sqrt(p*(1-p)/n + z*z/(4*n*n))/d
    return c - h, c + h
for k, n in ((3, 20), (30, 200), (300, 2000)):
    lo, hi = wilson(k, n); print(f"successes={k}/{n} rate={k/n:.3f} wilson95=[{lo:.3f}, {hi:.3f}]")
# five seeds of a synthetic defence evaluation: per-seed attack success rate
rates = []
for seed in range(5):
    r = random.Random(seed); rates.append(sum(r.random() < 0.12 for _ in range(200))/200)
m = sum(rates)/5; sd = math.sqrt(sum((x-m)**2 for x in rates)/4)
print("per-seed rates:", " ".join(f"{x:.3f}" for x in rates), f"mean={m:.3f} sd={sd:.3f}")
# paired bootstrap: baseline vs defence on the same 300 synthetic prompts
r = random.Random(42)
base = [r.random() < 0.30 for _ in range(300)]
defn = [b and r.random() < 0.6 for b in base]   # defence blocks ~40% of successes
diffs = []
for _ in range(2000):
    idx = [r.randrange(300) for _ in range(300)]
    diffs.append(sum(base[i] for i in idx)/300 - sum(defn[i] for i in idx)/300)
diffs.sort()
print(f"baseline_asr={sum(base)/300:.3f} defended_asr={sum(defn)/300:.3f} "
      f"reduction_ci95=[{diffs[50]:.3f}, {diffs[1949]:.3f}]")

Expected output:

successes=3/20 rate=0.150 wilson95=[0.052, 0.360]
successes=30/200 rate=0.150 wilson95=[0.107, 0.206]
successes=300/2000 rate=0.150 wilson95=[0.135, 0.166]
per-seed rates: 0.110 0.130 0.135 0.105 0.145 mean=0.125 sd=0.017
baseline_asr=0.350 defended_asr=0.190 reduction_ci95=[0.120, 0.203]

The lab shows why sample size matters: three successes in 20 and 300 in 2000 give the same rate, but the upper bound falls from 0.360 to 0.166. A threshold of 20% would fail the small sample and pass the large one, although the observed rate is identical. Rina therefore sized each Ask Lumen suite from the threshold before writing any cases, choosing the number of attempts that could support a pass if the defence worked as hoped.

27.3 End-to-end verification

Unit tests prove components; end-to-end tests prove the system. Replay adversarial suites through the full chain — retrieval, model, policy, tools in a sandbox — and inspect tool logs. Many failures appear only at integration points, such as identifiers being resolved after a policy check.

The Ask Lumen example of this was a cancellation that named the appointment by its position in a list ("the second one"); the policy service checked the patient, but the tool resolved "second" against a list fetched later, which could differ. Only an end-to-end replay revealed it, and the fix resolved the identifier before the check.

27.4 Reports

For each requirement: method, sample, result with interval, pass/fail, evidence location, versions tested and limitations.

A row in Rina's report for the cross-patient requirement read: adversarial suite, 400 cases, 0 failures, upper bound about 0.95%, pass, evidence in the staging log archive, model and prompt versions listed, English only. Each element lets a reviewer reproduce or challenge the result, and the limitation tells them where the claim stops. Rows for failed requirements use the same format, so a failure cannot be hidden in prose.

27.5 Regression

Every failure found becomes a permanent test. Rerun the suite on every model, prompt or tool change.

When the end-to-end replay found the cancellation identifier problem, Rina added that exact conversation to the permanent suite. Months later, a refactor of the cancellation tool briefly reintroduced the late lookup, and the regression test failed before the change reached production. Without the permanent test, the same flaw would have returned unnoticed, since nobody remembered the original incident in detail. The test is cheap to keep.

27.5b Verifying versions, not names

Verification applies to a specific configuration: model version, system prompt version, retrieval index snapshot, tool set and policy rules. A report that says "Ask Lumen passed" without versions cannot be relied on after the next change. Record the configuration hash in the report and have the deployment pipeline refuse configurations that have not been verified.

At LCH, a provider model update arrived on a Thursday, and the pipeline blocked deployment until the suite reran. Two refusals had changed wording, and all security requirements still passed, so the new version was recorded and released the next day.

27.6 Worked example (synthetic LCH case)

Rina verified ten requirements for Ask Lumen v2. Nine passed. "False clinic information ≤ 2% (95% upper bound)" failed: 4/300 observed (1.3%) but the Wilson upper bound was 3.4%. After the grounding check, 1/600 was observed with an upper bound below 1%, and the requirement passed.

The failure is instructive because the observed rate was well under the threshold; only the interval showed that 300 cases were not enough to be sure. After the grounding check, the lower failure rate and the larger sample together brought the upper bound to about 0.94%, comfortably under the requirement. Rina's report kept both results, so readers could see the requirement had failed before passing.

27.7 Errors and limits

A threshold chosen after the results can always be set to pass, which makes the test meaningless. Using the point estimate, as with 4 of 300, would have passed a requirement the evidence did not support. Unit tests alone would have missed the cancellation identifier flaw, which existed only where the policy service and tool met. Rina fixed thresholds in the test plan before any run, and the plan was reviewed.

Exercises

Synthesis

Verification proves requirements with pre-set statistical criteria and end-to-end tests, and keeps every failure as a regression test.

Continue studying — chapter 27 of 40

← Chapter 26 (draft) · Course chapter index · Chapter 28 (draft) →