OpenCyber Academy
Postgraduate · Chapter 6 · English
Course code: OCA-158Chapter 06: OCA-158-C006

Chapter 6 — AI and machine-learning security: controlled evaluation

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 06 published. Map, example, EX026.

flowchart TD
  N0["Learning outcomes"]
  N1["6.1 Controlled comparison"]
  N2["6.2 Metrics and sample size"]
  N3["6.3 Classical threats"]
  N4["6.4 Reporting for decisions"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX026"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["6.1 Controlled comparison"]
  N2["6.2 Metrics and sample size"]
  N3["6.3 Classical threats"]
  N4["6.4 Reporting for decisions"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX026"]

[Learning outcomes]
    |
    v
[6.1 Controlled comparison]
    |
    v
[6.2 Metrics and sample size]
    |
    v
[6.3 Classical threats]
    |
    v
[6.4 Reporting for decisions]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX026]

Learning outcomes

You will (1) design a controlled evaluation comparing defences; (2) control confounders such as suite overlap and model versions; (3) choose effect metrics and sample sizes; (4) evaluate classical ML threats such as poisoning; and (5) report results that support a decision.

6.1 Controlled comparison

Change one thing at a time. Hold the model version, prompt, retrieval index and test suite fixed while comparing defences; record versions. Split suites into development cases (used to tune defences) and held-out cases (used once for reporting), and include adaptive cases written by people who did not build the defence.

At Corvane, the comparison of three defences used the same model version, prompts and 400 held-out cases. Record the suite version and who wrote the adaptive cases, so a later rerun can use the same conditions. Keep the held-out cases sealed until the final comparison.

6.2 Metrics and sample size

Measure effects: harmful tool calls, records disclosed, money moved. Also measure costs: false refusals and task completion on normal traffic. To distinguish 15% from 10% attack success with reasonable confidence you need a few hundred cases per arm; to claim a rate below 1% you need several hundred cases with zero or near-zero failures.

Amira's evaluation used 400 adaptive cases and 2,000 normal conversations, enough to separate rates of 22%, 9% and 0.5% and to measure task completion. Plan sample sizes before running tests, not after seeing the results.

6.3 Classical threats

For predictive models, controlled evaluation includes poisoning and evasion. The lab shows label-flip poisoning barely moving clean accuracy, while a backdoor with 80 triggered points reaches 98% attack success at unchanged clean accuracy. Clean-accuracy monitoring alone would miss it.

"""Lab P: label-flip poisoning and a backdoor trigger on a toy nearest-centroid
classifier (synthetic data, stdlib only). Teaching model only."""
import random
rng = random.Random(11)
def point(label, trigger=False):
    cx = 1.0 if label else -1.0
    x = [rng.gauss(cx, 0.8), rng.gauss(cx, 0.8), 0.0]
    if trigger: x[2] = 5.0          # third feature is the 'trigger'
    return x, label
clean = [point(i % 2) for i in range(400)]
test = [point(i % 2) for i in range(400)]
def fit(data):
    c = {}
    for lab in (0, 1):
        pts = [x for x, y in data if y == lab]
        c[lab] = [sum(p[i] for p in pts)/len(pts) for i in range(3)]
    return c
def pred(c, x):
    d = {k: sum((x[i]-v[i])**2 for i in range(3)) for k, v in c.items()}
    return min(d, key=d.get)
def accuracy(c, data): return sum(pred(c, x) == y for x, y in data)/len(data)
print("label-flip poisoning (flip a fraction of class-1 labels to 0):")
for frac in (0.0, 0.1, 0.2, 0.3, 0.4):
    data = [(x, y) for x, y in clean]; ones = [i for i, (x, y) in enumerate(data) if y == 1]
    for i in ones[:int(frac*len(ones))]: data[i] = (data[i][0], 0)
    print(f"  flipped={frac:.1f} clean_test_accuracy={accuracy(fit(data), test):.3f}")
print("backdoor: add n triggered class-1 points labelled 0")
trig_test = [point(1, trigger=True) for _ in range(200)]
for n in (0, 10, 40, 80):
    data = clean + [(point(1, trigger=True)[0], 0) for _ in range(n)]
    c = fit(data)
    asr = sum(pred(c, x) == 0 for x, _ in trig_test)/len(trig_test)
    print(f"  poison_points={n:3d} clean_accuracy={accuracy(c, test):.3f} attack_success={asr:.3f}")

Expected output:

label-flip poisoning (flip a fraction of class-1 labels to 0):
  flipped=0.0 clean_test_accuracy=0.965
  flipped=0.1 clean_test_accuracy=0.965
  flipped=0.2 clean_test_accuracy=0.953
  flipped=0.3 clean_test_accuracy=0.950
  flipped=0.4 clean_test_accuracy=0.945
backdoor: add n triggered class-1 points labelled 0
  poison_points=  0 clean_accuracy=0.965 attack_success=0.075
  poison_points= 10 clean_accuracy=0.963 attack_success=0.145
  poison_points= 40 clean_accuracy=0.963 attack_success=0.685
  poison_points= 80 clean_accuracy=0.968 attack_success=0.980

For FraudScore, Amira's team adds a backdoor test with synthetic trigger invoices, because a poisoned training set might pass clean-accuracy checks. Report both clean accuracy and trigger success for each test. Run the trigger test after every retraining, since a new data source can introduce a backdoor at any time.

6.4 Reporting for decisions

Report per defence: attack success with interval, difference from baseline with interval, side effects, cost, and conditions (versions, suites, dates). End with what the result implies for the decision and what it does not show.

At Corvane, the report on the three defences ended with a recommendation for privilege separation and stated that the classifier would remain only as a monitoring signal. It also stated what the result did not show: behaviour on future model versions or new attack styles.

6.5 Worked example (synthetic Corvane case)

Amira's team compares the baseline Copilot, Copilot with an injection classifier, and Copilot with privilege separation, on 400 held-out adaptive cases plus 2,000 normal conversations. Harmful tool calls: 22%, 9% and 0.5%. Normal-task completion: 91%, 86% and 90%. The classifier cut attacks but also blocked legitimate carrier emails containing words like "ignore the previous slot". She recommends privilege separation, keeping the classifier only as a monitoring signal. For FraudScore she adds a backdoor test using synthetic trigger invoices.

The result is reported with completion rates as well as attack rates, so the cost of each defence is clear.

6.6 Errors and limits

  • Changing model and defence together.
  • Tuning on the reporting suite.
  • Ignoring false refusals.

Each error makes a comparison unreliable. Changing model and defence together means nobody knows which caused the change. Tuning on the reporting suite makes the defence look better than it is on new attacks. Ignoring false refusals hides costs, such as the classifier blocking legitimate carrier emails at Corvane. An evaluation plan with one of these flaws should be corrected before the results reach a decision owner.

Exercises

  • EX026 — Design. Design a controlled evaluation of two RAG access-control approaches. Rubric: 3 = fixed variables, suites, metrics, sample size; 2 = three; 1 = one or two; 0 = none.
  • EX027 — Confounders. List four confounders and how to control them. Rubric: 3 = four with controls; 2 = three; 1 = one or two; 0 = none.
  • EX028 — Poison lab. Interpret the backdoor rows of the lab. Rubric: 3 = attack success rises with points, clean accuracy flat, monitoring implication; 2 = two; 1 = one; 0 = wrong.
  • EX029 — Trade-off. Should Corvane keep the classifier in the blocking path? Justify. Rubric: 3 = uses both attack and completion results; 2 = one; 1 = assertion; 0 = none.
  • EX030 — Report. Write the results paragraph for §6.5. Rubric: 3 = rates, intervals or sizes, side effects, conditions, implication; 2 = four; 1 = two; 0 = none.

Synthesis

Controlled evaluation changes one thing at a time, uses held-out adaptive suites, measures effects and side effects, and reports in a form that drives the decision.

Continue studying — chapter 06 of 12

← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →

Worked answers

EX026. Fix the model, prompt, index and documents. Compare approach 1 (filtering after retrieval) with approach 2 (an index partitioned by permission). Use 300 held-out cross-user queries, including adaptive ones, plus 1,000 normal queries. Metrics: unauthorised passages returned, with an interval, and answer quality. Repeat across three seeds.

EX027. - Model version drift: pin the version. - Suite overlap with tuning data: use held-out suites. - Tester knowledge: use an independent adaptive team. - Time-of-day load affecting timeouts: run the arms interleaved.

EX028. As poison points rise from 0 to 80, attack success rises from 0.075 to 0.980, while clean accuracy stays between 0.963 and 0.968. Monitoring clean accuracy would not detect the backdoor, so trigger tests or data provenance checks are needed.

EX029. No, not in the blocking path. It cut attacks (22% to 9%) but lowered completion (91% to 86%). Privilege separation achieves 0.5% with 90% completion, so the classifier adds cost without benefit there. Keep it as a monitoring signal.

EX030. "On 400 held-out adaptive cases and 2,000 normal conversations (model v3, October 2026), harmful tool calls were 22% for the baseline, 9% with the classifier and 0.5% with privilege separation. Normal completion was 91%, 86% and 90%. Privilege separation reduces attacks without hurting service; the classifier blocks legitimate carrier text. These results do not cover other languages."