◇ OpenCyber Academy
Postgraduate · Chapter 3 · English

Chapter 3 — AI and machine-learning security: specialist evidence

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 3 — AI and machine-learning security: specialist evidence

Lesson map

Lesson 03 published. Map, example, EX011.

flowchart TD
  N0["Learning outcomes"]
  N1["3.1 Kinds of evidence"]
  N2["3.2 Rates and intervals"]
  N3["3.3 Weak evaluations"]
  N4["3.4 Privacy and poisoning evidence"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX011"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["3.1 Kinds of evidence"]
  N2["3.2 Rates and intervals"]
  N3["3.3 Weak evaluations"]
  N4["3.4 Privacy and poisoning evidence"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX011"]

[Learning outcomes]
    |
    v
[3.1 Kinds of evidence]
    |
    v
[3.2 Rates and intervals]
    |
    v
[3.3 Weak evaluations]
    |
    v
[3.4 Privacy and poisoning evidence]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX011]

Learning outcomes

You will (1) distinguish kinds of AI-security evidence and their strength; (2) report attack-success rates with confidence intervals; (3) recognise weak evaluations, including non-adaptive attacks; (4) interpret privacy and poisoning measurements; and (5) write an evidence statement a reviewer can check.

3.1 Kinds of evidence

From weaker to stronger: vendor claims; documentation of controls; configuration review; non-adaptive testing with fixed suites; adaptive testing by people who know the defence; independent red teaming; and monitored production outcomes. Each has a place, but a claim should cite evidence matching its strength. "The model resists injection" needs adaptive testing; "the agent cannot refund over €20" can be shown by configuration review plus tests of the policy service, because enforcement does not depend on the model.

Amira's evidence statement in the worked example below credits configuration and tests of the policy service.

3.2 Rates and intervals

Report successes over trials with a 95% interval (Wilson is a good default). Three successes in 20 trials (15%) gives roughly 5–36%; the same rate in 2,000 trials gives 13.5–16.6%. Compare defences on the same suite and report the difference with an interval. Vary random seeds and report the spread.

"""Lab S: reporting attack-success rates honestly: Wilson intervals,
seed variation and a paired bootstrap. Synthetic trial outcomes only."""
import math, random
def wilson(k, n, z=1.96):
    p = k/n; d = 1 + z*z/n
    c = (p + z*z/(2*n))/d; h = z*math.sqrt(p*(1-p)/n + z*z/(4*n*n))/d
    return c - h, c + h
for k, n in ((3, 20), (30, 200), (300, 2000)):
    lo, hi = wilson(k, n); print(f"successes={k}/{n} rate={k/n:.3f} wilson95=[{lo:.3f}, {hi:.3f}]")
# five seeds of a synthetic defence evaluation: per-seed attack success rate
rates = []
for seed in range(5):
    r = random.Random(seed); rates.append(sum(r.random() < 0.12 for _ in range(200))/200)
m = sum(rates)/5; sd = math.sqrt(sum((x-m)**2 for x in rates)/4)
print("per-seed rates:", " ".join(f"{x:.3f}" for x in rates), f"mean={m:.3f} sd={sd:.3f}")
# paired bootstrap: baseline vs defence on the same 300 synthetic prompts
r = random.Random(42)
base = [r.random() < 0.30 for _ in range(300)]
defn = [b and r.random() < 0.6 for b in base]   # defence blocks ~40% of successes
diffs = []
for _ in range(2000):
    idx = [r.randrange(300) for _ in range(300)]
    diffs.append(sum(base[i] for i in idx)/300 - sum(defn[i] for i in idx)/300)
diffs.sort()
print(f"baseline_asr={sum(base)/300:.3f} defended_asr={sum(defn)/300:.3f} "
      f"reduction_ci95=[{diffs[50]:.3f}, {diffs[1949]:.3f}]")

Expected output:

successes=3/20 rate=0.150 wilson95=[0.052, 0.360]
successes=30/200 rate=0.150 wilson95=[0.107, 0.206]
successes=300/2000 rate=0.150 wilson95=[0.135, 0.166]
per-seed rates: 0.110 0.130 0.135 0.105 0.145 mean=0.125 sd=0.017
baseline_asr=0.350 defended_asr=0.190 reduction_ci95=[0.120, 0.203]

Amira's 41 of 300 result is reported this way, with an interval of about 10–18%, so the board sees both the rate and its uncertainty. Report the number of trials beside every rate, and say whether the cases were adaptive.

3.3 Weak evaluations

Carlini et al. (arXiv:1902.06705) list common faults in robustness evaluations; Athalye et al. (arXiv:1802.00420) showed that defences relying on obfuscated gradients were broken by adapted attacks. Warning signs: only fixed public attack strings; the defender wrote all tests; no baseline; tiny samples; success measured by the model's words rather than by what happened (a tool call, a data release).

The vendor's "99%" claim in the worked example below shows several of these signs at once: public strings only, a vendor-written suite and success measured by classifier output.

3.4 Privacy and poisoning evidence

Membership inference (Shokri et al., arXiv:1610.05820) is measured as attack advantage or AUC over balanced member/non-member sets; differential privacy (Abadi et al., arXiv:1607.00133) gives a formal bound instead of a measurement. Poisoning evidence reports both clean accuracy and attack success, because backdoors (Gu et al., arXiv:1708.06733) can leave clean accuracy unchanged.

For FraudScore, poisoning evidence would mean testing synthetic trigger invoices as well as clean accuracy, as Amira's team does in the evaluation chapter. A single clean-accuracy figure would not reveal a backdoor.

3.5 Worked example (synthetic Corvane case)

A vendor claims its injection filter "blocks 99% of attacks". Amira asks for the suite: 500 public strings, non-adaptive, measured by classifier output. Her team builds 300 adaptive cases against Dispatch Copilot and measures tool calls: the filter alone allows 41 harmful calls (13.7%, interval ≈ 10.2–18.0%). With the refund policy service, 0 of 300 reach payment (upper bound ≈ 1%). Her evidence statement credits the policy service, not the filter.

The difference between 13.7% and 0 of 300 shows why measuring effects matters: the filter changes text, the policy service changes outcomes. Amira keeps the filter only as a signal.

3.6 Errors and limits

Each error leads readers to trust the wrong control. A rate without trials or intervals cannot be compared or checked. Non-adaptive results say little about attackers who know the defence; the vendor's 99% fell to 13.7% when Amira's team adapted. Measuring the model's words rather than tool calls misses harm that happens anyway. Review each evidence statement against these three points and revise it before it is reported.

Exercises

Synthesis

Specialist evidence is graded by strength, reported with intervals, gathered adaptively and measured by effects. Credit claims to the controls that the evidence actually supports.

Continue studying — chapter 03 of 12

← Chapter 02 (draft) · Course chapter index · Chapter 04 (draft) →

Worked answers

EX011. From weakest to strongest: 1. The vendor brochure. 2. The design document of controls. 3. Configuration review of the policy service. 4. Fixed-suite tests. 5. Adaptive red-team results with tool-call measurement.

EX012. 3/20 has a wide interval (about 5–36%), so it supports only "probably under ~36%". 300/2,000 narrows to about 13.5–16.6% and supports precise comparisons.

EX013. It uses only public strings, so it is non-adaptive. It measures classifier output rather than effects. It reports no interval or false-refusal rate. The vendor did the evaluating, so it is not independent.

EX014. A backdoor can leave clean accuracy unchanged while attack success is high. The lab shows 0.968 clean accuracy alongside 0.980 attack success, so either number alone misleads.

EX015. "In 300 held-out adaptive conversations on an isolated copy (model v3, prompt p12, October 2026), no refund outside policy reached payment; the 95% upper bound is about 1%. This depends on the policy service, not the model or filter."