Chapter 24 — AI and machine-learning security: methods I
Lesson map
Lesson 24 published. Map, example, EX070.
flowchart TD
N0["Learning outcomes"]
N1["24.1 Evasion evaluation"]
N2["24.2 Poisoning evaluation"]
N3["24.3 Membership-inference evaluation"]
N4["24.4 From toys to real components"]
N5["24.4b Reading results responsibly"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX070"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["24.1 Evasion evaluation"]
N2["24.2 Poisoning evaluation"]
N3["24.3 Membership-inference evaluation"]
N4["24.4 From toys to real components"]
N5["24.4b Reading results responsibly"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX070"]
[Learning outcomes]
|
v
[24.1 Evasion evaluation]
|
v
[24.2 Poisoning evaluation]
|
v
[24.3 Membership-inference evaluation]
|
v
[24.4 From toys to real components]
|
v
[24.4b Reading results responsibly]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX070]
Learning outcomes
You will (1) run basic adversarial evaluations — evasion, poisoning and membership inference — on toy models; (2) interpret results as curves rather than single numbers; (3) recognise the limits of toy evaluations; and (4) plan the transfer of methods to real components under authorization.
24.1 Evasion evaluation
Evasion tests perturb inputs within a budget and measure accuracy. Report accuracy as a function of budget. The lab trains a logistic regression and applies FGSM-style perturbations; accuracy drops from 98.5% to chance as ε grows.
"""Lab E: evasion of a toy linear classifier (synthetic data, stdlib only).
Trains logistic regression on two synthetic clusters, then applies a
fast-gradient-sign (FGSM-style) perturbation of size eps to every test point
and reports clean vs adversarial accuracy. Teaching model only."""
import math, random
rng = random.Random(7)
def point(label):
cx = 1.0 if label else -1.0
return [rng.gauss(cx, 0.7), rng.gauss(cx, 0.7)], label
train = [point(i % 2) for i in range(400)]
test = [point(i % 2) for i in range(200)]
w, b = [0.0, 0.0], 0.0
sig = lambda z: 1 / (1 + math.exp(-z))
for epoch in range(200):
gw, gb = [0.0, 0.0], 0.0
for x, y in train:
p = sig(w[0]*x[0] + w[1]*x[1] + b)
gw[0] += (p - y)*x[0]; gw[1] += (p - y)*x[1]; gb += p - y
w = [w[i] - 0.1*gw[i]/len(train) for i in range(2)]; b -= 0.1*gb/len(train)
def acc(data, eps):
ok = 0
for x, y in data:
# gradient of the loss w.r.t. x is (p - y) * w; FGSM steps along its sign
p = sig(w[0]*x[0] + w[1]*x[1] + b)
xa = [x[i] + eps*math.copysign(1, (p - y)*w[i]) for i in range(2)]
ok += (sig(w[0]*xa[0] + w[1]*xa[1] + b) >= 0.5) == bool(y)
return ok / len(data)
print(f"weights w=({w[0]:.3f}, {w[1]:.3f}) b={b:.3f}")
for eps in (0.0, 0.25, 0.5, 0.75, 1.0):
print(f"eps={eps:.2f} accuracy={acc(test, eps):.3f}")
Expected output:
weights w=(1.795, 1.756) b=-0.080
eps=0.00 accuracy=0.985
eps=0.25 accuracy=0.950
eps=0.50 accuracy=0.840
eps=0.75 accuracy=0.660
eps=1.00 accuracy=0.505
The curve tells a fuller story than any single point. At ε = 0.25 the model still holds 95%, but by 0.75 it is down to 66%, and at 1.0 it is guessing. Whether that matters depends on how large a change an attacker can realistically make, which is why the budget is set from the threat model.
24.2 Poisoning evaluation
Poisoning tests add or relabel training data and measure both clean accuracy and targeted effects. Backdoor success can be high while clean accuracy is unchanged, so both metrics are essential.
"""Lab P: label-flip poisoning and a backdoor trigger on a toy nearest-centroid
classifier (synthetic data, stdlib only). Teaching model only."""
import random
rng = random.Random(11)
def point(label, trigger=False):
cx = 1.0 if label else -1.0
x = [rng.gauss(cx, 0.8), rng.gauss(cx, 0.8), 0.0]
if trigger: x[2] = 5.0 # third feature is the 'trigger'
return x, label
clean = [point(i % 2) for i in range(400)]
test = [point(i % 2) for i in range(400)]
def fit(data):
c = {}
for lab in (0, 1):
pts = [x for x, y in data if y == lab]
c[lab] = [sum(p[i] for p in pts)/len(pts) for i in range(3)]
return c
def pred(c, x):
d = {k: sum((x[i]-v[i])**2 for i in range(3)) for k, v in c.items()}
return min(d, key=d.get)
def accuracy(c, data): return sum(pred(c, x) == y for x, y in data)/len(data)
print("label-flip poisoning (flip a fraction of class-1 labels to 0):")
for frac in (0.0, 0.1, 0.2, 0.3, 0.4):
data = [(x, y) for x, y in clean]; ones = [i for i, (x, y) in enumerate(data) if y == 1]
for i in ones[:int(frac*len(ones))]: data[i] = (data[i][0], 0)
print(f" flipped={frac:.1f} clean_test_accuracy={accuracy(fit(data), test):.3f}")
print("backdoor: add n triggered class-1 points labelled 0")
trig_test = [point(1, trigger=True) for _ in range(200)]
for n in (0, 10, 40, 80):
data = clean + [(point(1, trigger=True)[0], 0) for _ in range(n)]
c = fit(data)
asr = sum(pred(c, x) == 0 for x, _ in trig_test)/len(trig_test)
print(f" poison_points={n:3d} clean_accuracy={accuracy(c, test):.3f} attack_success={asr:.3f}")
Expected output:
label-flip poisoning (flip a fraction of class-1 labels to 0):
flipped=0.0 clean_test_accuracy=0.965
flipped=0.1 clean_test_accuracy=0.965
flipped=0.2 clean_test_accuracy=0.953
flipped=0.3 clean_test_accuracy=0.950
flipped=0.4 clean_test_accuracy=0.945
backdoor: add n triggered class-1 points labelled 0
poison_points= 0 clean_accuracy=0.965 attack_success=0.075
poison_points= 10 clean_accuracy=0.963 attack_success=0.145
poison_points= 40 clean_accuracy=0.963 attack_success=0.685
poison_points= 80 clean_accuracy=0.968 attack_success=0.980
The output shows both effects. Flipping up to 40% of class-1 labels costs only two points of clean accuracy, because the centroid moves slowly. The backdoor is subtler: with 80 triggered points, clean accuracy stays at 0.968 while 98% of triggered inputs are misclassified. A test that checked only clean accuracy would pass this poisoned model.
24.3 Membership-inference evaluation
Privacy tests measure how well an attacker can distinguish training members from non-members, as AUC and as true-positive rate at low false-positive rate.
"""Lab M: membership inference against an overfit 1-nearest-neighbour 'model'
that returns a confidence score. Synthetic records only; teaching model."""
import random, math
rng = random.Random(5)
def rec():
y = rng.random() < 0.5
return [rng.gauss(0.6 if y else -0.6, 1.0) for _ in range(4)], int(y)
members = [rec() for _ in range(150)]; nonmembers = [rec() for _ in range(150)]
def confidence(x, y, k):
# k-NN vote share for the true label, computed over the training set
d = sorted((sum((a-b)**2 for a, b in zip(x, m)), my) for m, my in members)
return sum(1 for _, my in d[:k] if my == y)/k
def auc(pos, neg):
s = sum((p > n) + 0.5*(p == n) for p in pos for n in neg)
return s/(len(pos)*len(neg))
for k in (1, 5, 25):
pos = [confidence(x, y, k) for x, y in members]
neg = [confidence(x, y, k) for x, y in nonmembers]
best = max(sum(p >= t for p in pos)/len(pos) - sum(n >= t for n in neg)/len(neg)
for t in sorted(set(pos + neg)))
print(f"k={k:2d} attack_auc={auc(pos, neg):.3f} best_advantage={best:.3f}")
Expected output:
k= 1 attack_auc=0.600 best_advantage=0.200
k= 5 attack_auc=0.521 best_advantage=0.073
k=25 attack_auc=0.520 best_advantage=0.060
The lab shows overfitting as the cause: the 1-nearest-neighbour model memorises each training point, giving an attack AUC of 0.600 and an advantage of 0.200. Averaging over five neighbours already brings AUC down to 0.521, and 25 neighbours change little more, so the attacker ends barely better than guessing. AUC alone can still hide risk: an attack that identifies a few members with near certainty matters even when the average looks harmless, which is why the true-positive rate at a low false-positive rate is reported too.
24.4 From toys to real components
Toy models make concepts measurable in seconds; real models need stronger attacks (multi-step PGD per Madry et al., arXiv:1706.06083; adaptive attacks per Carlini et al., arXiv:1902.06705), larger test sets and authorised environments. Always evaluate the deployed artefact. Never run these methods against systems you are not authorised to test.
The labs teach what the metrics mean; they do not show how a real model will behave. Real evaluations also need budgets justified by the threat model and enough items for narrow intervals.
24.4b Reading results responsibly
Each evaluation answers a narrow question: "how does this model behave under this attack with this budget?" Record the attack, its parameters, the number of test items, seeds and intervals. A strong result under a weak attack says little; a weak result under a strong attack is informative. When a defence appears to work, try an attack that knows the defence before believing it — the central lesson of Athalye et al. (arXiv:1802.00420). Keep all evaluation code and outputs so that later model versions can be compared on exactly the same tests.
24.5 Worked example (synthetic NRS case)
Sam ran the three labs to brief NRS managers in fifteen minutes: evasion shows sensitivity to small changes, poisoning shows why the labelling contractor matters, and membership inference shows why Pulse's sensor data should not be exposed with full confidences. Sam then scoped a real evaluation of TrackEye in the test yard with printed markings, with the operations director's written authorization.
Using toy models for the briefing kept attention on concepts rather than on TrackEye's actual numbers, which did not yet exist.
24.6 Errors and limits
- Reporting one ε point.
- Measuring only clean accuracy after poisoning.
- Presenting toy results as evidence about real systems.
A single ε value hides how quickly performance falls and invites cherry-picking the budget that looks best. Measuring only clean accuracy after poisoning misses backdoors entirely, as the lab's 80-point row shows. Toy results describe toy models; presenting them as evidence about TrackEye or Wrench would mislead decision-makers. Sam labels every toy result as illustrative on the slide where it appears. The label stays when slides are reused.
Exercises
- EX070 — Evasion curve. From the lab, what accuracy is reported at ε = 0.25 and 1.00? Rubric: 3 = 0.950 and 0.505 with interpretation; 2 = numbers; 1 = one; 0 = wrong.
- EX071 — Poison metrics. Why must poisoning evaluations report both clean accuracy and attack success? Rubric: 3 = backdoors keep clean accuracy; 0.963 vs 0.685 at 40 points; 2 = reason only; 1 = vague; 0 = wrong.
- EX072 — Toy limits. Give two reasons the labs' results do not transfer directly to TrackEye. Rubric: 3 = model complexity/nonlinearity, physical conditions, stronger attacks needed, data differences; 2 = one; 1 = vague; 0 = none.
Synthesis
Basic adversarial evaluation methods — evasion curves, dual-metric poisoning tests, low-FPR privacy tests — build intuition fast; real assurance requires stronger attacks on deployed artefacts under authorization.
Continue studying — chapter 24 of 50
← Chapter 23 (draft) · Course chapter index · Chapter 25 (draft) →