Chapter 15 — AI and machine-learning security: guided lab
Lesson map
Lesson 15 published. Map, example, EX043.
flowchart TD
N0["Learning outcomes"]
N1["15.1 Lab rules"]
N2["15.2 Evasion"]
N3["15.3 Poisoning and backdoors"]
N4["15.4 Membership inference"]
N5["15.5 Extraction"]
N6["15.5b Writing up a lab"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX043"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["15.1 Lab rules"]
N2["15.2 Evasion"]
N3["15.3 Poisoning and backdoors"]
N4["15.4 Membership inference"]
N5["15.5 Extraction"]
N6["15.5b Writing up a lab"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX043"]
[Learning outcomes]
|
v
[15.1 Lab rules]
|
v
[15.2 Evasion]
|
v
[15.3 Poisoning and backdoors]
|
v
[15.4 Membership inference]
|
v
[15.5 Extraction]
|
v
[15.5b Writing up a lab]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX043]
Learning outcomes
You will (1) run contained labs on evasion, poisoning, membership inference and extraction; (2) interpret their outputs; (3) connect each to a real-world control; and (4) follow lab safety rules.
15.1 Lab rules
Labs run offline on synthetic data with the Python 3 standard library. They are toy models that make concepts measurable; they are not evidence about production systems and not tools for attacking real ones. Jonas uses each lab to understand an attack class on ReturnGuard and to justify a control, not to estimate how ReturnGuard itself would fare. A reader should treat every lab number as a property of the toy model and its synthetic data. Fixed seeds mean each lab prints the same output on every run.
15.2 Evasion
"""Lab E: evasion of a toy linear classifier (synthetic data, stdlib only).
Trains logistic regression on two synthetic clusters, then applies a
fast-gradient-sign (FGSM-style) perturbation of size eps to every test point
and reports clean vs adversarial accuracy. Teaching model only."""
import math, random
rng = random.Random(7)
def point(label):
cx = 1.0 if label else -1.0
return [rng.gauss(cx, 0.7), rng.gauss(cx, 0.7)], label
train = [point(i % 2) for i in range(400)]
test = [point(i % 2) for i in range(200)]
w, b = [0.0, 0.0], 0.0
sig = lambda z: 1 / (1 + math.exp(-z))
for epoch in range(200):
gw, gb = [0.0, 0.0], 0.0
for x, y in train:
p = sig(w[0]*x[0] + w[1]*x[1] + b)
gw[0] += (p - y)*x[0]; gw[1] += (p - y)*x[1]; gb += p - y
w = [w[i] - 0.1*gw[i]/len(train) for i in range(2)]; b -= 0.1*gb/len(train)
def acc(data, eps):
ok = 0
for x, y in data:
# gradient of the loss w.r.t. x is (p - y) * w; FGSM steps along its sign
p = sig(w[0]*x[0] + w[1]*x[1] + b)
xa = [x[i] + eps*math.copysign(1, (p - y)*w[i]) for i in range(2)]
ok += (sig(w[0]*xa[0] + w[1]*xa[1] + b) >= 0.5) == bool(y)
return ok / len(data)
print(f"weights w=({w[0]:.3f}, {w[1]:.3f}) b={b:.3f}")
for eps in (0.0, 0.25, 0.5, 0.75, 1.0):
print(f"eps={eps:.2f} accuracy={acc(test, eps):.3f}")
Expected output:
weights w=(1.795, 1.756) b=-0.080
eps=0.00 accuracy=0.985
eps=0.25 accuracy=0.950
eps=0.50 accuracy=0.840
eps=0.75 accuracy=0.660
eps=1.00 accuracy=0.505
Accuracy falls from 0.985 to 0.505 as the perturbation budget grows. Control: do not let a single model score make irreversible decisions; add rules and human review. For ReturnGuard, this supports Jonas's recommendation that a high fraud score leads to human review rather than an automatic refusal. For ReturnGuard, the features a fraudster can actually change, such as return reason and timing, are fewer and coarser than the lab's free perturbation, so the lab shows the direction of the risk rather than its size. The size is measured instead by the Chapter 14 evasion tests, which change only the reason, the timing and the split of real fraud cases.
15.3 Poisoning and backdoors
"""Lab P: label-flip poisoning and a backdoor trigger on a toy nearest-centroid
classifier (synthetic data, stdlib only). Teaching model only."""
import random
rng = random.Random(11)
def point(label, trigger=False):
cx = 1.0 if label else -1.0
x = [rng.gauss(cx, 0.8), rng.gauss(cx, 0.8), 0.0]
if trigger: x[2] = 5.0 # third feature is the 'trigger'
return x, label
clean = [point(i % 2) for i in range(400)]
test = [point(i % 2) for i in range(400)]
def fit(data):
c = {}
for lab in (0, 1):
pts = [x for x, y in data if y == lab]
c[lab] = [sum(p[i] for p in pts)/len(pts) for i in range(3)]
return c
def pred(c, x):
d = {k: sum((x[i]-v[i])**2 for i in range(3)) for k, v in c.items()}
return min(d, key=d.get)
def accuracy(c, data): return sum(pred(c, x) == y for x, y in data)/len(data)
print("label-flip poisoning (flip a fraction of class-1 labels to 0):")
for frac in (0.0, 0.1, 0.2, 0.3, 0.4):
data = [(x, y) for x, y in clean]; ones = [i for i, (x, y) in enumerate(data) if y == 1]
for i in ones[:int(frac*len(ones))]: data[i] = (data[i][0], 0)
print(f" flipped={frac:.1f} clean_test_accuracy={accuracy(fit(data), test):.3f}")
print("backdoor: add n triggered class-1 points labelled 0")
trig_test = [point(1, trigger=True) for _ in range(200)]
for n in (0, 10, 40, 80):
data = clean + [(point(1, trigger=True)[0], 0) for _ in range(n)]
c = fit(data)
asr = sum(pred(c, x) == 0 for x, _ in trig_test)/len(trig_test)
print(f" poison_points={n:3d} clean_accuracy={accuracy(c, test):.3f} attack_success={asr:.3f}")
Expected output:
label-flip poisoning (flip a fraction of class-1 labels to 0):
flipped=0.0 clean_test_accuracy=0.965
flipped=0.1 clean_test_accuracy=0.965
flipped=0.2 clean_test_accuracy=0.953
flipped=0.3 clean_test_accuracy=0.950
flipped=0.4 clean_test_accuracy=0.945
backdoor: add n triggered class-1 points labelled 0
poison_points= 0 clean_accuracy=0.965 attack_success=0.075
poison_points= 10 clean_accuracy=0.963 attack_success=0.145
poison_points= 40 clean_accuracy=0.963 attack_success=0.685
poison_points= 80 clean_accuracy=0.968 attack_success=0.980
Label flipping barely moves clean accuracy; 80 backdoor points reach 98% attack success with unchanged clean accuracy. Control: data provenance, restricted labelling rights, trigger tests. For ReturnGuard, whose labels come from staff, this explains why Jonas recommended that only managers can relabel returns: a few planted labels with a trigger could leave clean accuracy unchanged while letting matching returns through. Restricting relabelling to managers and logging every label change are what make such a trigger hard to plant without anyone noticing.
15.4 Membership inference
"""Lab M: membership inference against an overfit 1-nearest-neighbour 'model'
that returns a confidence score. Synthetic records only; teaching model."""
import random, math
rng = random.Random(5)
def rec():
y = rng.random() < 0.5
return [rng.gauss(0.6 if y else -0.6, 1.0) for _ in range(4)], int(y)
members = [rec() for _ in range(150)]; nonmembers = [rec() for _ in range(150)]
def confidence(x, y, k):
# k-NN vote share for the true label, computed over the training set
d = sorted((sum((a-b)**2 for a, b in zip(x, m)), my) for m, my in members)
return sum(1 for _, my in d[:k] if my == y)/k
def auc(pos, neg):
s = sum((p > n) + 0.5*(p == n) for p in pos for n in neg)
return s/(len(pos)*len(neg))
for k in (1, 5, 25):
pos = [confidence(x, y, k) for x, y in members]
neg = [confidence(x, y, k) for x, y in nonmembers]
best = max(sum(p >= t for p in pos)/len(pos) - sum(n >= t for n in neg)/len(neg)
for t in sorted(set(pos + neg)))
print(f"k={k:2d} attack_auc={auc(pos, neg):.3f} best_advantage={best:.3f}")
Expected output:
k= 1 attack_auc=0.600 best_advantage=0.200
k= 5 attack_auc=0.521 best_advantage=0.073
k=25 attack_auc=0.520 best_advantage=0.060
When the model's score comes from a single nearest neighbour (k=1), the attack's advantage is 0.20; it falls as k grows in this setup. Control: limit output detail, test for leakage, consider differential privacy (Abadi et al., arXiv:1607.00133). For ReturnGuard, the lesson is to limit what the scoring service reveals: a label is safer than a detailed score. The lab's setup is small, so the falling advantage is a property of this toy, not a general law. ReturnGuard now returns only review or accept to the store front, not a score.
15.5 Extraction
"""Lab X: model extraction of a toy linear scoring API by querying it.
The 'API' returns a real-valued score; d+1 well-chosen queries recover the
weights exactly, and rounding the output limits fidelity. Synthetic model."""
import random
rng = random.Random(9)
d = 5
secret_w = [round(rng.uniform(-2, 2), 3) for _ in range(d)]; secret_b = 0.42
def api(x, decimals=None):
s = sum(a*b for a, b in zip(secret_w, x)) + secret_b
return round(s, decimals) if decimals is not None else s
for dec in (None, 2, 0):
b = api([0]*d, dec)
w = [api([1 if j == i else 0 for j in range(d)], dec) - b for i in range(d)]
err = max(abs(a-c) for a, c in zip(w, secret_w))
test = [[rng.uniform(-1, 1) for _ in range(d)] for _ in range(500)]
agree = sum((sum(a*c for a, c in zip(w, x))+b > 0) == (api(x) > 0) for x in test)/500
print(f"output_rounding={str(dec):4s} queries={d+1} max_weight_error={err:.3f} label_agreement={agree:.3f}")
Expected output:
output_rounding=None queries=6 max_weight_error=0.000 label_agreement=1.000
output_rounding=2 queries=6 max_weight_error=0.004 label_agreement=0.998
output_rounding=0 queries=6 max_weight_error=0.534 label_agreement=0.882
With exact scores, six queries recover the model exactly; rounding to whole numbers raises error to 0.534. Control: return labels or coarse scores, rate limit, monitor query patterns (Tramèr et al., arXiv:1609.02943). This is the lab behind Jonas's advice that ReturnGuard return only review or accept to the store front, and that scoring calls be rate-limited per account. The partner key later seen varying one feature at a time matched the pattern this lab describes. Rounding alone did not make extraction impossible in the lab, which is why rate limits accompany it.
15.5b Writing up a lab
For each lab, record: what was varied, what was measured, the result table, one sentence on what it shows, one sentence on what it does not show, and the control it suggests. This habit transfers directly to real evaluations, where the same structure keeps reports honest and readable for colleagues who were not in the lab. Jonas's write-up of the extraction lab ends with what it does not show: whether ReturnGuard's real scoring API could be copied, since the lab's model is linear and tiny.
15.6 Worked example (synthetic Brightwell case)
Jonas runs the four labs and writes one control recommendation each for ReturnGuard: human review for refusals; only managers can relabel returns; return only "review/accept" not a score to the store front; rate-limit the scoring API per staff account. Each recommendation comes from one lab: evasion to human review, poisoning to labelling rights, membership inference and extraction to coarse outputs and rate limits. None claims ReturnGuard has the weakness; each limits what a weakness would cost. Each control fits a part of ReturnGuard that Brightwell owns.
15.7 Errors and limits
- Treating toy results as production numbers.
- Running attack code against real systems.
Jonas's write-ups avoid both by stating, for each lab, what it does not show about ReturnGuard, and by running every lab offline on synthetic data. A toy result quoted as a production figure would mislead owners in the same way an untested claim would, and attack code pointed at a real service would break the authorisation every test depends on. The lab rules make both explicit before anyone runs code.
Exercises
- EX043 — Interpret. Explain one number from each lab. Rubric: 3 = four correct interpretations; 2 = three; 1 = one or two; 0 = none.
- EX044 — Controls. Match each lab to a control for ReturnGuard. Rubric: 3 = four apt; 2 = three; 1 = one or two; 0 = none.
- EX045 — Limits. Why are these labs not evidence about Brightwell's real model? Rubric: 3 = toy data, different model, different threat model; 2 = two; 1 = one; 0 = none.
Synthesis
Contained labs make abstract attacks measurable and each points to a practical control.
Continue studying — chapter 15 of 30
← Chapter 14 (draft) · Course chapter index · Chapter 16 (draft) →