◇ OpenCyber Academy
Technical · Chapter 11 · English

Chapter 11 — AI and machine-learning security: verification

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 11 — AI and machine-learning security: verification

Lesson map

Lesson 11 published. Map, example, EX041.

flowchart TD
  N0["Learning outcomes"]
  N1["11.1 Verify, don't assume"]
  N2["11.2 What to verify"]
  N3["11.3 Re-testing after changes"]
  N4["11.4 Reporting with uncertainty"]
  N5["11.5 Records"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX041"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["11.1 Verify, don't assume"]
  N2["11.2 What to verify"]
  N3["11.3 Re-testing after changes"]
  N4["11.4 Reporting with uncertainty"]
  N5["11.5 Records"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX041"]

[Learning outcomes]
    |
    v
[11.1 Verify, don't assume]
    |
    v
[11.2 What to verify]
    |
    v
[11.3 Re-testing after changes]
    |
    v
[11.4 Reporting with uncertainty]
    |
    v
[11.5 Records]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX041]

Learning outcomes

You will (1) verify that configured controls work; (2) re-test after changes; (3) report verification results with uncertainty; (4) keep verification records.

11.1 Verify, don't assume

A configured control may not work: a rule may have a typo, a setting may not have been deployed, a new tool may bypass the check. Verification means running tests that would fail if the control did not work. A control that has never been seen to block something is a control you hope works.

At Kestrel, the ownership check passed code review but first reached staging with the comparison reversed for one router type, so it allowed exactly the restarts it should block on those routers. Only a negative test caught it. The Wi-Fi password tool is the other example: added two weeks after Ana's configuration, it called the router-management service directly and so bypassed the restart tool's check entirely. Verification is the step that finds these before customers do.

11.2 What to verify

For each control, at least one positive test (normal use still works) and one negative test (the bad action is blocked). Example: the restart check allows a customer to restart their own router and blocks restarts of others' routers. Use held-out injection tests that were not used to tune the configuration.

For the ticket tool, the positive test is a synthetic customer opening a normal-priority ticket about their own line; the negative tests are an injected request for an urgent ticket and a ticket raised against another account. Both must be graded from the ticket log. The held-out set matters because Ana tuned the configuration against her 40 injection tests; only the 20 held-out tests, run once, show whether the fix generalises.

11.3 Re-testing after changes

Re-run the suite when the model version, prompt, retrieval sources, tools or configuration change. Automate this in the deployment pipeline so it cannot be forgotten.

Kestrel's pipeline runs the full HelpBot suite on staging for every change to the model version, the system prompt, the retrieval collections, the tools or the policy configuration, and blocks deployment on any cross-customer action or canary leak. That is how the Wi-Fi password tool's missing ownership check was caught before release. Vendor model updates count as changes even though Kestrel did not write them; the version is pinned until the suite passes on the new one.

11.4 Reporting with uncertainty

Report counts and intervals: 0 failures in 60 tests means the true failure rate is probably under about 5%, not zero. More tests narrow the interval.

"""Lab S: reporting attack-success rates honestly: Wilson intervals,
seed variation and a paired bootstrap. Synthetic trial outcomes only."""
import math, random
def wilson(k, n, z=1.96):
    p = k/n; d = 1 + z*z/n
    c = (p + z*z/(2*n))/d; h = z*math.sqrt(p*(1-p)/n + z*z/(4*n*n))/d
    return c - h, c + h
for k, n in ((3, 20), (30, 200), (300, 2000)):
    lo, hi = wilson(k, n); print(f"successes={k}/{n} rate={k/n:.3f} wilson95=[{lo:.3f}, {hi:.3f}]")
# five seeds of a synthetic defence evaluation: per-seed attack success rate
rates = []
for seed in range(5):
    r = random.Random(seed); rates.append(sum(r.random() < 0.12 for _ in range(200))/200)
m = sum(rates)/5; sd = math.sqrt(sum((x-m)**2 for x in rates)/4)
print("per-seed rates:", " ".join(f"{x:.3f}" for x in rates), f"mean={m:.3f} sd={sd:.3f}")
# paired bootstrap: baseline vs defence on the same 300 synthetic prompts
r = random.Random(42)
base = [r.random() < 0.30 for _ in range(300)]
defn = [b and r.random() < 0.6 for b in base]   # defence blocks ~40% of successes
diffs = []
for _ in range(2000):
    idx = [r.randrange(300) for _ in range(300)]
    diffs.append(sum(base[i] for i in idx)/300 - sum(defn[i] for i in idx)/300)
diffs.sort()
print(f"baseline_asr={sum(base)/300:.3f} defended_asr={sum(defn)/300:.3f} "
      f"reduction_ci95=[{diffs[50]:.3f}, {diffs[1949]:.3f}]")

Expected output:

successes=3/20 rate=0.150 wilson95=[0.052, 0.360]
successes=30/200 rate=0.150 wilson95=[0.107, 0.206]
successes=300/2000 rate=0.150 wilson95=[0.135, 0.166]
per-seed rates: 0.110 0.130 0.135 0.105 0.145 mean=0.125 sd=0.017
baseline_asr=0.350 defended_asr=0.190 reduction_ci95=[0.120, 0.203]

The lab shows why intervals matter. Three successes in 20 trials gives an interval between 5.2% and 36%; the same rate over 2,000 trials narrows to 13.5%–16.6%. Seed variation adds its own spread: five seeds of the same synthetic evaluation gave rates between 0.105 and 0.145. The paired bootstrap estimates the defence's reduction at 0.120 to 0.203. Ana's reports therefore state "0 of 60 (upper bound about 6%)" rather than "fixed".

For 0 of 30, the bound is about 10%.

11.5 Records

Keep a verification record: control, test IDs, versions, date, results, who ran them. Auditors and future technicians rely on these records.

Ana's record for the restart ownership check reads: control, tool-service ownership rule; tests, 60 cross-account restart attempts plus positive own-router tests; versions of the orchestrator, tool service and pinned model; date; result, 0 of 60 succeeded and own-router restarts all worked; tester, Ana, reviewed by the team lead. When the Wi-Fi tool was added, this record showed the auditor exactly which tests the new tool lacked.

The record sits beside RST-N01–N30 in version control, so the tests and their evidence change together.

11.6 Worked example (synthetic Kestrel case)

Two weeks after Ana's configuration, a developer adds a "change Wi-Fi password" tool. The pipeline re-runs the suite: positive tests pass, but a negative test shows the new tool lacks the ownership check — a customer can change another customer's Wi-Fi password. Deployment is blocked; the check is added; the suite passes, 0 of 30 cross-customer attempts (upper bound about 10%). Ana adds 60 more tests to narrow it.

The record from this run went into the verification log beside the restart check's. The pipeline did its job because the suite included negative tests for every tool that touches a router; a suite of only positive tests would have passed the new tool, since customers could change their own Wi-Fi passwords. With 90 attempts and 0 successes, the bound tightens to about 4%.

11.7 Errors and limits

Each error matches something Ana saw. Treating configuration as proof would have shipped the ownership check with its reversed comparison. A suite of only negative tests passes a system that refuses everything, including the 50 normal tasks customers actually need. No re-test after adding a tool is how the Wi-Fi password tool almost reached customers without an ownership check. Each mistake leaves a record that says "verified" while the control does not work.

Exercises

Synthesis

Verification turns configuration into evidence: positive and negative tests, re-run on every change, reported with uncertainty and recorded.

Continue studying — chapter 11 of 18

← Chapter 10 (draft) · Course chapter index · Chapter 12 (draft) →