Chapter 4 — AI and machine-learning security: competing explanations
Lesson map
Lesson 04 published. Map, example, EX021.
flowchart TD
N0["Learning outcomes"]
N1["4.1 Why competing explanations matter"]
N2["4.2 A catalogue of alternatives in AI security"]
N3["4.3 Ablations and controls"]
N4["4.4 Reasoning about mechanisms with the labs"]
N5["4.6 Practice activities"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX021"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["4.1 Why competing explanations matter"]
N2["4.2 A catalogue of alternatives in AI security"]
N3["4.3 Ablations and controls"]
N4["4.4 Reasoning about mechanisms with the labs"]
N5["4.6 Practice activities"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX021"]
[Learning outcomes]
|
v
[4.1 Why competing explanations matter]
|
v
[4.2 A catalogue of alternatives in AI security]
|
v
[4.3 Ablations and controls]
|
v
[4.4 Reasoning about mechanisms with the labs]
|
v
[4.6 Practice activities]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX021]
Learning outcomes
You will (1) list alternative explanations for an observed security effect; (2) design ablations and controls that separate them; (3) recognise confounders typical of AI-security experiments (overfitting, data leakage, distribution shift, evaluation artefacts, weak attacks); (4) use the poisoning and evasion labs to reason about mechanisms; and (5) state conclusions at the level the evidence supports.
4.1 Why competing explanations matter
An observed effect — "deduplication lowered membership-inference advantage" — can have several causes. Deduplication might reduce memorisation of repeated records specifically; it might simply reduce effective training-set size and therefore change overfitting; it might change class balance; or the attack might be weaker on the new model for reasons unrelated to privacy. A dissertation that names and tests these alternatives is far stronger than one that reports the effect alone. Leo's seminar discussion raised the overfitting explanation before he had any data.
4.2 A catalogue of alternatives in AI security
Six alternatives recur. Generalisation vs. specific mechanism: privacy leakage correlates with the train–test gap, so compare treatments at matched generalisation gaps, or include a control that changes data size without deduplicating. Leakage: duplicates across train and test splits inflate accuracy and confuse privacy metrics (Kapoor and Narayanan, arXiv:2207.07048). Distribution shift: if members and non-members differ in distribution, attacks detect shift, not membership. Weak or mis-tuned attack: a defence "works" because the attack was not adapted (Athalye et al., arXiv:1802.00420); use a stronger or calibrated attack as a control. Evaluation artefacts: metric choice or thresholds tuned on the test set. Randomness: a single seed can produce an apparent effect.
4.3 Ablations and controls
An ablation removes or varies one component to see its contribution. Leo's plan adds a fourth condition: random removal of the same number of records that deduplication would remove. If random removal gives the same privacy gain, the explanation is dataset size; if deduplication gives more, the duplicates matter. He also adds a control attack calibrated with reference models, which is known to be stronger at low FPR than a simple confidence threshold. Write the expected result of each condition before running.
4.4 Reasoning about mechanisms with the labs
The poisoning lab gives a clean contrast. Label flipping of up to 40% of one class barely moves a nearest-centroid model (0.965 → 0.945), because centroids average many points; a backdoor using a separate trigger feature reaches 98% attack success with 80 points while clean accuracy stays at 0.968. The mechanism — a new, otherwise unused feature direction — explains why clean metrics miss it. A competing explanation ("the model is just inaccurate") is ruled out by the unchanged clean accuracy. For Leo, the analogy is that duplicated records may give the model a specific route to memorise them.
"""Lab P: label-flip poisoning and a backdoor trigger on a toy nearest-centroid
classifier (synthetic data, stdlib only). Teaching model only."""
import random
rng = random.Random(11)
def point(label, trigger=False):
cx = 1.0 if label else -1.0
x = [rng.gauss(cx, 0.8), rng.gauss(cx, 0.8), 0.0]
if trigger: x[2] = 5.0 # third feature is the 'trigger'
return x, label
clean = [point(i % 2) for i in range(400)]
test = [point(i % 2) for i in range(400)]
def fit(data):
c = {}
for lab in (0, 1):
pts = [x for x, y in data if y == lab]
c[lab] = [sum(p[i] for p in pts)/len(pts) for i in range(3)]
return c
def pred(c, x):
d = {k: sum((x[i]-v[i])**2 for i in range(3)) for k, v in c.items()}
return min(d, key=d.get)
def accuracy(c, data): return sum(pred(c, x) == y for x, y in data)/len(data)
print("label-flip poisoning (flip a fraction of class-1 labels to 0):")
for frac in (0.0, 0.1, 0.2, 0.3, 0.4):
data = [(x, y) for x, y in clean]; ones = [i for i, (x, y) in enumerate(data) if y == 1]
for i in ones[:int(frac*len(ones))]: data[i] = (data[i][0], 0)
print(f" flipped={frac:.1f} clean_test_accuracy={accuracy(fit(data), test):.3f}")
print("backdoor: add n triggered class-1 points labelled 0")
trig_test = [point(1, trigger=True) for _ in range(200)]
for n in (0, 10, 40, 80):
data = clean + [(point(1, trigger=True)[0], 0) for _ in range(n)]
c = fit(data)
asr = sum(pred(c, x) == 0 for x, _ in trig_test)/len(trig_test)
print(f" poison_points={n:3d} clean_accuracy={accuracy(c, test):.3f} attack_success={asr:.3f}")
Expected output:
label-flip poisoning (flip a fraction of class-1 labels to 0):
flipped=0.0 clean_test_accuracy=0.965
flipped=0.1 clean_test_accuracy=0.965
flipped=0.2 clean_test_accuracy=0.953
flipped=0.3 clean_test_accuracy=0.950
flipped=0.4 clean_test_accuracy=0.945
backdoor: add n triggered class-1 points labelled 0
poison_points= 0 clean_accuracy=0.965 attack_success=0.075
poison_points= 10 clean_accuracy=0.963 attack_success=0.145
poison_points= 40 clean_accuracy=0.963 attack_success=0.685
poison_points= 80 clean_accuracy=0.968 attack_success=0.980
4.5 Worked example (synthetic Harbourline case)
In pilot data under the calibrated attack, deduplication reduced TPR at 1% FPR from 6.1% to 3.4%. Random removal of the same number of records reduced it to 5.6%. The difference between deduplication and random removal (2.2 points) is the part plausibly attributable to removing duplicates; the remaining 0.5 points is explained by size. The simple confidence-threshold attack sat near its floor in every condition (the pilot in Chapter 5 describes this), so only the calibrated attack could separate them. Leo's conclusion sentence changed from "deduplication protects privacy" to "most of the observed reduction in membership leakage is attributable to removing duplicated records rather than to the smaller training set, under the evaluated attack".
4.6 Practice activities
- List five competing explanations for your expected result. 2. Design one ablation and one control for each of the two most plausible. 3. Rewrite your expected conclusion at the evidence level you will have. For the list, use the catalogue above and add any explanation specific to your system. For the ablations, change one thing at a time and say what result would support each explanation. For the conclusion, write the sentence you could defend if the main effect appeared, and compare it with Leo's revised sentence, which names the attack and what was ruled out.
4.7 Errors and limits
Four errors recur, and each lets a competing explanation survive; check your plan for all four:
- Reporting an effect without testing alternatives; an effect without alternatives tested may be size, shift or chance.
- Ablations that change several things at once; such an ablation cannot attribute the effect.
- Using only a weak attack to evaluate a privacy defence; a weak attack can make any defence look good, which is why Leo added the calibrated attack.
- Claiming mechanism from correlation alone; correlation, such as leakage tracking duplication, does not by itself show mechanism without controls like random removal.
Exercises
- EX021 — Alternatives. List four competing explanations for "a guardrail reduced jailbreak success". Rubric: 3 = four plausible (weaker attack set, refusals over-triggered, distribution change, seed/judge artefact); 2 = three; 1 = one or two; 0 = none.
- EX022 — Ablation. Design the random-removal control and explain what each outcome would mean. Rubric: 3 = same count removed at random; equal gain → size, larger gain → duplicates; 2 = design only; 1 = vague; 0 = none.
- EX023 — Lab mechanism. Why does label flipping barely change the centroid model while the backdoor succeeds? Rubric: 3 = centroids average many points vs trigger uses a new feature direction unaffected by clean data; 2 = partial; 1 = vague; 0 = wrong.
- EX024 — Lab numbers. Report clean accuracy and attack success at 80 poison points. Rubric: 3 = 0.968 and 0.980; 2 = one; 1 = approximate; 0 = wrong.
- EX025 — Stronger attack. Why include a calibrated attack as a control? Rubric: 3 = ensures defence effect is not an artefact of a weak attack, especially at low FPR; 2 = partial; 1 = vague; 0 = wrong.
- EX026 — Conclusion level. Rewrite "dedup fixes privacy" at the evidence level of 4.5. Rubric: 3 = attributes portion, names attack and conditions; 2 = partial; 1 = still overclaims; 0 = unchanged.
Synthesis
Effects need explanations, and explanations need tests. Ablations, controls and stronger attacks separate mechanisms from artefacts, and the resulting conclusions are narrower but defensible.
Continue studying — chapter 04 of 08
← Chapter 03 (draft) · Course chapter index · Chapter 05 (draft) →