Chapter 25 — Laboratory I
Lesson map
Lesson 25 published. Map, example, EX073.
flowchart TD
N0["Learning outcomes"]
N1["25.1 Lab rules"]
N2["25.2 Experiments"]
N3["25.3 Interpreting"]
N4["25.4 Lab report"]
N5["25.4b Changing parameters"]
N6["25.4c Safety of lab code"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX073"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["25.1 Lab rules"]
N2["25.2 Experiments"]
N3["25.3 Interpreting"]
N4["25.4 Lab report"]
N5["25.4b Changing parameters"]
N6["25.4c Safety of lab code"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX073"]
[Learning outcomes]
|
v
[25.1 Lab rules]
|
v
[25.2 Experiments]
|
v
[25.3 Interpreting]
|
v
[25.4 Lab report]
|
v
[25.4b Changing parameters]
|
v
[25.4c Safety of lab code]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX073]
Learning outcomes
You will (1) set up a safe, isolated AI-security lab; (2) run evasion, poisoning, privacy and injection experiments on toy models; (3) record results reproducibly; and (4) write a short lab report.
25.1 Lab rules
Use isolated environments, synthetic data and harmless payloads (for example addresses on the reserved .invalid domain). Never point lab tools at real services. Record what you ran, with versions and seeds.
The reserved domain matters because a payload that sends data to a real address could leak information even in a test. Recording versions and seeds lets Rina reproduce the exact table later, and lets her supervisor check that the numbers in her report came from the code she submitted rather than from an earlier edit.
25.2 Experiments
Run the four labs and record their outputs:
"""Lab E: evasion of a toy linear classifier (synthetic data, stdlib only).
Trains logistic regression on two synthetic clusters, then applies a
fast-gradient-sign (FGSM-style) perturbation of size eps to every test point
and reports clean vs adversarial accuracy. Teaching model only."""
import math, random
rng = random.Random(7)
def point(label):
cx = 1.0 if label else -1.0
return [rng.gauss(cx, 0.7), rng.gauss(cx, 0.7)], label
train = [point(i % 2) for i in range(400)]
test = [point(i % 2) for i in range(200)]
w, b = [0.0, 0.0], 0.0
sig = lambda z: 1 / (1 + math.exp(-z))
for epoch in range(200):
gw, gb = [0.0, 0.0], 0.0
for x, y in train:
p = sig(w[0]*x[0] + w[1]*x[1] + b)
gw[0] += (p - y)*x[0]; gw[1] += (p - y)*x[1]; gb += p - y
w = [w[i] - 0.1*gw[i]/len(train) for i in range(2)]; b -= 0.1*gb/len(train)
def acc(data, eps):
ok = 0
for x, y in data:
# gradient of the loss w.r.t. x is (p - y) * w; FGSM steps along its sign
p = sig(w[0]*x[0] + w[1]*x[1] + b)
xa = [x[i] + eps*math.copysign(1, (p - y)*w[i]) for i in range(2)]
ok += (sig(w[0]*xa[0] + w[1]*xa[1] + b) >= 0.5) == bool(y)
return ok / len(data)
print(f"weights w=({w[0]:.3f}, {w[1]:.3f}) b={b:.3f}")
for eps in (0.0, 0.25, 0.5, 0.75, 1.0):
print(f"eps={eps:.2f} accuracy={acc(test, eps):.3f}")
Expected output:
weights w=(1.795, 1.756) b=-0.080
eps=0.00 accuracy=0.985
eps=0.25 accuracy=0.950
eps=0.50 accuracy=0.840
eps=0.75 accuracy=0.660
eps=1.00 accuracy=0.505
"""Lab P: label-flip poisoning and a backdoor trigger on a toy nearest-centroid
classifier (synthetic data, stdlib only). Teaching model only."""
import random
rng = random.Random(11)
def point(label, trigger=False):
cx = 1.0 if label else -1.0
x = [rng.gauss(cx, 0.8), rng.gauss(cx, 0.8), 0.0]
if trigger: x[2] = 5.0 # third feature is the 'trigger'
return x, label
clean = [point(i % 2) for i in range(400)]
test = [point(i % 2) for i in range(400)]
def fit(data):
c = {}
for lab in (0, 1):
pts = [x for x, y in data if y == lab]
c[lab] = [sum(p[i] for p in pts)/len(pts) for i in range(3)]
return c
def pred(c, x):
d = {k: sum((x[i]-v[i])**2 for i in range(3)) for k, v in c.items()}
return min(d, key=d.get)
def accuracy(c, data): return sum(pred(c, x) == y for x, y in data)/len(data)
print("label-flip poisoning (flip a fraction of class-1 labels to 0):")
for frac in (0.0, 0.1, 0.2, 0.3, 0.4):
data = [(x, y) for x, y in clean]; ones = [i for i, (x, y) in enumerate(data) if y == 1]
for i in ones[:int(frac*len(ones))]: data[i] = (data[i][0], 0)
print(f" flipped={frac:.1f} clean_test_accuracy={accuracy(fit(data), test):.3f}")
print("backdoor: add n triggered class-1 points labelled 0")
trig_test = [point(1, trigger=True) for _ in range(200)]
for n in (0, 10, 40, 80):
data = clean + [(point(1, trigger=True)[0], 0) for _ in range(n)]
c = fit(data)
asr = sum(pred(c, x) == 0 for x, _ in trig_test)/len(trig_test)
print(f" poison_points={n:3d} clean_accuracy={accuracy(c, test):.3f} attack_success={asr:.3f}")
Expected output:
label-flip poisoning (flip a fraction of class-1 labels to 0):
flipped=0.0 clean_test_accuracy=0.965
flipped=0.1 clean_test_accuracy=0.965
flipped=0.2 clean_test_accuracy=0.953
flipped=0.3 clean_test_accuracy=0.950
flipped=0.4 clean_test_accuracy=0.945
backdoor: add n triggered class-1 points labelled 0
poison_points= 0 clean_accuracy=0.965 attack_success=0.075
poison_points= 10 clean_accuracy=0.963 attack_success=0.145
poison_points= 40 clean_accuracy=0.963 attack_success=0.685
poison_points= 80 clean_accuracy=0.968 attack_success=0.980
"""Lab M: membership inference against an overfit 1-nearest-neighbour 'model'
that returns a confidence score. Synthetic records only; teaching model."""
import random, math
rng = random.Random(5)
def rec():
y = rng.random() < 0.5
return [rng.gauss(0.6 if y else -0.6, 1.0) for _ in range(4)], int(y)
members = [rec() for _ in range(150)]; nonmembers = [rec() for _ in range(150)]
def confidence(x, y, k):
# k-NN vote share for the true label, computed over the training set
d = sorted((sum((a-b)**2 for a, b in zip(x, m)), my) for m, my in members)
return sum(1 for _, my in d[:k] if my == y)/k
def auc(pos, neg):
s = sum((p > n) + 0.5*(p == n) for p in pos for n in neg)
return s/(len(pos)*len(neg))
for k in (1, 5, 25):
pos = [confidence(x, y, k) for x, y in members]
neg = [confidence(x, y, k) for x, y in nonmembers]
best = max(sum(p >= t for p in pos)/len(pos) - sum(n >= t for n in neg)/len(neg)
for t in sorted(set(pos + neg)))
print(f"k={k:2d} attack_auc={auc(pos, neg):.3f} best_advantage={best:.3f}")
Expected output:
k= 1 attack_auc=0.600 best_advantage=0.200
k= 5 attack_auc=0.521 best_advantage=0.073
k=25 attack_auc=0.520 best_advantage=0.060
"""Lab X: model extraction of a toy linear scoring API by querying it.
The 'API' returns a real-valued score; d+1 well-chosen queries recover the
weights exactly, and rounding the output limits fidelity. Synthetic model."""
import random
rng = random.Random(9)
d = 5
secret_w = [round(rng.uniform(-2, 2), 3) for _ in range(d)]; secret_b = 0.42
def api(x, decimals=None):
s = sum(a*b for a, b in zip(secret_w, x)) + secret_b
return round(s, decimals) if decimals is not None else s
for dec in (None, 2, 0):
b = api([0]*d, dec)
w = [api([1 if j == i else 0 for j in range(d)], dec) - b for i in range(d)]
err = max(abs(a-c) for a, c in zip(w, secret_w))
test = [[rng.uniform(-1, 1) for _ in range(d)] for _ in range(500)]
agree = sum((sum(a*c for a, c in zip(w, x))+b > 0) == (api(x) > 0) for x in test)/500
print(f"output_rounding={str(dec):4s} queries={d+1} max_weight_error={err:.3f} label_agreement={agree:.3f}")
Expected output:
output_rounding=None queries=6 max_weight_error=0.000 label_agreement=1.000
output_rounding=2 queries=6 max_weight_error=0.004 label_agreement=0.998
output_rounding=0 queries=6 max_weight_error=0.534 label_agreement=0.882
Rina ran all four on her laptop with the stated seeds and compared each line with the expected output before writing anything. All matched. When a classmate's evasion run differed in the third decimal place, the cause was a different Python version, which they recorded in the setup notes. Checking against expected output first catches such differences before they reach a report or a marker.
25.3 Interpreting
Evasion: accuracy falls as the budget grows. Poisoning: label flipping barely matters for this model, but a backdoor reaches 98% attack success with clean accuracy unchanged. Membership: a memorising model leaks; a smoother one leaks little. Extraction: a linear API is copied exactly with six queries; rounding outputs reduces but does not eliminate fidelity.
These results carry different lessons. Evasion and extraction depend on what the attacker can query; poisoning depends on who controls training data; membership depends on how much the model memorises. Each points to a different control at LCH: query limits and coarse outputs for NoShow, label provenance for DocSort.
25.4 Lab report
Structure: aim, setup (versions, seeds), method, results (tables), interpretation, limitations (toy models, synthetic data), and what you would test next on a real system with authorization.
Rina's report stated in its limitations that the poisoning lab uses a tiny linear model, so the 80-point figure does not predict how many letters would be needed to backdoor DocSort. What carries over is the pattern: attack success can be high while clean accuracy stays unchanged, so clean testing alone cannot detect a backdoor.
25.4b Changing parameters
A lab becomes an experiment when you change one parameter and predict the effect before running. Try changing the evasion budget list, the number of backdoor points or the k values in the membership lab; write your prediction, run, and compare. Record each run's parameters and output. This habit — predict, run, compare, record — is the core of experimental work in later chapters and in the capstone.
Rina predicted that 20 backdoor points would give attack success between the 10-point and 40-point results, then ran it and recorded the value beside her prediction.
25.4c Safety of lab code
Lab code that implements attacks should stay in the lab. Do not adapt it to target real services, and do not share modified versions that point at real endpoints. The toy models here are deliberately simple so that concepts are visible; their attacks are not effective tools against real systems, and that is intentional.
Rina's repository marks the lab folder as teaching material and keeps no endpoint configuration in it. Payload addresses use the reserved domain, so even a copied script cannot send data anywhere real.
25.5 Worked example (synthetic LCH case)
Rina's lab report connected each toy result to an LCH system: evasion to DocSort's crafted letters, poisoning to contractor labels, membership to NoShow's confidence outputs and extraction to a planned partner API. Her "next steps" proposed authorised tests on LCH's staging copies.
LCH's security lead approved the staging tests with a written scope covering which systems, which dates and which payloads could be used. The extraction result was the most persuasive, because the planned partner API returned exact scores; the team agreed to round outputs before launch, accepting the loss in fidelity shown in the lab.
25.6 Errors and limits
- Presenting toy results as facts about real systems.
- Missing versions and seeds.
- Lab tools pointed at real endpoints.
A toy model with five features tells us about mechanisms, not about DocSort's real robustness, so results must be labelled as teaching results. Missing versions and seeds make a result impossible to reproduce or check. Pointing lab code at real services, even by accident, can harm patients and breaks the rules under which the lab is allowed; Rina's lab config contains only local addresses.
Exercises
- EX073 — Extraction. How many queries recover the linear model exactly, and why? Rubric: 3 = 6 (d + 1): one for bias, one per weight; 2 = number; 1 = vague; 0 = wrong.
- EX074 — Interpretation. Why does label flipping barely change the toy model? Rubric: 3 = centroids average many points; minority flips shift little; 2 = partial; 1 = vague; 0 = wrong.
- EX075 — Report. List the seven parts of the lab report. Rubric: 3 = all seven; 2 = five; 1 = three; 0 = none.
Synthesis
Laboratory I gives hands-on intuition for the four attack families and the habit of reproducible, honest lab reports.
Continue studying — chapter 25 of 40
← Chapter 24 (draft) · Course chapter index · Chapter 26 (draft) →