Chapter 22 — AI and machine-learning security: data and dependencies
Lesson map
Lesson 22 published. Map, example, EX064.
flowchart TD
N0["Learning outcomes"]
N1["22.1 Data risks"]
N2["22.2 Supply chain"]
N3["22.3 Documentation"]
N4["22.4 Provenance and integrity"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX064"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["22.1 Data risks"]
N2["22.2 Supply chain"]
N3["22.3 Documentation"]
N4["22.4 Provenance and integrity"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX064"]
[Learning outcomes]
|
v
[22.1 Data risks]
|
v
[22.2 Supply chain]
|
v
[22.3 Documentation]
|
v
[22.4 Provenance and integrity]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX064]
Learning outcomes
You will (1) assess training, fine-tuning and retrieval data for poisoning and privacy risk; (2) manage third-party models, datasets and libraries as supply-chain dependencies; (3) document data with datasheets and models with model cards; and (4) design provenance and integrity controls.
22.1 Data risks
Training data can be poisoned to degrade performance or implant backdoors; it can contain personal data that the model may leak; it can be biased. Retrieval data can carry injected instructions and must be access-controlled. The lab shows that a few dozen triggered points create a strong backdoor while clean accuracy stays high.
"""Lab P: label-flip poisoning and a backdoor trigger on a toy nearest-centroid
classifier (synthetic data, stdlib only). Teaching model only."""
import random
rng = random.Random(11)
def point(label, trigger=False):
cx = 1.0 if label else -1.0
x = [rng.gauss(cx, 0.8), rng.gauss(cx, 0.8), 0.0]
if trigger: x[2] = 5.0 # third feature is the 'trigger'
return x, label
clean = [point(i % 2) for i in range(400)]
test = [point(i % 2) for i in range(400)]
def fit(data):
c = {}
for lab in (0, 1):
pts = [x for x, y in data if y == lab]
c[lab] = [sum(p[i] for p in pts)/len(pts) for i in range(3)]
return c
def pred(c, x):
d = {k: sum((x[i]-v[i])**2 for i in range(3)) for k, v in c.items()}
return min(d, key=d.get)
def accuracy(c, data): return sum(pred(c, x) == y for x, y in data)/len(data)
print("label-flip poisoning (flip a fraction of class-1 labels to 0):")
for frac in (0.0, 0.1, 0.2, 0.3, 0.4):
data = [(x, y) for x, y in clean]; ones = [i for i, (x, y) in enumerate(data) if y == 1]
for i in ones[:int(frac*len(ones))]: data[i] = (data[i][0], 0)
print(f" flipped={frac:.1f} clean_test_accuracy={accuracy(fit(data), test):.3f}")
print("backdoor: add n triggered class-1 points labelled 0")
trig_test = [point(1, trigger=True) for _ in range(200)]
for n in (0, 10, 40, 80):
data = clean + [(point(1, trigger=True)[0], 0) for _ in range(n)]
c = fit(data)
asr = sum(pred(c, x) == 0 for x, _ in trig_test)/len(trig_test)
print(f" poison_points={n:3d} clean_accuracy={accuracy(c, test):.3f} attack_success={asr:.3f}")
Expected output:
label-flip poisoning (flip a fraction of class-1 labels to 0):
flipped=0.0 clean_test_accuracy=0.965
flipped=0.1 clean_test_accuracy=0.965
flipped=0.2 clean_test_accuracy=0.953
flipped=0.3 clean_test_accuracy=0.950
flipped=0.4 clean_test_accuracy=0.945
backdoor: add n triggered class-1 points labelled 0
poison_points= 0 clean_accuracy=0.965 attack_success=0.075
poison_points= 10 clean_accuracy=0.963 attack_success=0.145
poison_points= 40 clean_accuracy=0.963 attack_success=0.685
poison_points= 80 clean_accuracy=0.968 attack_success=0.980
The two lab tables tell different stories. Flipping 40% of labels lowers clean accuracy only from 0.965 to 0.945, so noisy labelling is a nuisance; 80 triggered points give 0.980 attack success, so targeted poisoning is a threat.
22.2 Supply chain
Pre-trained models, datasets, libraries and plug-ins come from third parties. OWASP LLM 2025 lists supply chain (LLM03) and data and model poisoning (LLM04). Controls: trusted sources, pinned versions, hash and signature verification, safe file formats, scanning, and evaluation before use.
DocSort's vendor model is LCH's largest supply-chain dependency, and the signature check described in the cryptography chapter covers it. Ask Lumen's libraries are pinned to versions in a lock file, and updates go through review. LCH avoids model file formats that can execute code when loaded, which removes one route by which a tampered download could run on its servers.
22.3 Documentation
Datasheets for datasets (Gebru et al., arXiv:1803.09010) record motivation, composition, collection, preprocessing and recommended uses. Model cards (Mitchell et al., arXiv:1810.03993) record intended use, evaluation, limitations and ethical considerations. Both help others judge whether a dependency is safe for their use.
Rina wrote a datasheet for DocSort's training letters. Writing it exposed that nobody knew which months of letters were included or how ambiguous cases were labelled. The model card she requested from the vendor stated the model was evaluated on letters from a different health system, which LCH recorded as a limitation and checked with its own test set.
22.4 Provenance and integrity
Record where each data item came from and who changed it; version datasets; hash content at collection; review samples from new sources; restrict who can label.
For DocSort, this means each training batch records the contractor who labelled it, the date and a hash of the labelled file. If a trigger is later found, LCH can find which batches contain it and who labelled them, retrain without them, and decide whether the contractor arrangement needs review. Without that record, the whole training set would be suspect.
22.5 Worked example (synthetic LCH case)
DocSort was retrained monthly on letters labelled by a temporary contractor team. Rina found no record of who labelled what. She introduced per-batch provenance, a 5% expert review sample per batch and an automated check for rare words strongly associated with one route — a simple trigger detector. In the first month it flagged a harmless template phrase used by one GP practice, which was documented as benign.
The false alarm was useful: it showed the detector worked and taught the team what benign triggers look like.
22.6 Errors and limits
- Unknown labellers and unversioned data.
- Downloading models without verification.
- Datasheets and model cards never written.
Without labeller records, a poisoned batch cannot be traced or removed, which was DocSort's situation before Rina's changes. Unverified downloads let a tampered model reach production with no warning. Missing datasheets and model cards mean nobody can judge whether a dataset or model fits a new use, so mismatches, such as DocSort's evaluation on another health system's letters, go unnoticed until results disappoint in production and patients receive misrouted letters.
Exercises
- EX064 — Lab reading. With 40 poison points, what are clean accuracy and attack success? Rubric: 3 = 0.963 and 0.685; 2 = one; 1 = approximate; 0 = wrong.
- EX065 — Supply-chain controls. List four controls for a downloaded pre-trained model. Rubric: 3 = trusted source, pinned version, hash/signature, safe format, evaluation; 2 = three; 1 = one or two; 0 = none.
- EX066 — Trigger detector. Why did a benign phrase get flagged, and what should happen? Rubric: 3 = rare word strongly linked to a route; review and document as benign (false positive); 2 = partial; 1 = vague; 0 = wrong.
Synthesis
Data and dependencies shape what models learn and do. Provenance, verification, documentation and review keep poisoned or untrusted inputs from silently becoming behaviour.
Continue studying — chapter 22 of 40
← Chapter 21 (draft) · Course chapter index · Chapter 23 (draft) →