Chapter 20 — AI and machine-learning security: foundational cases
Lesson map
Lesson 20 published. Map, example, EX058.
flowchart TD
N0["Learning outcomes"]
N1["20.1 Adversarial examples"]
N2["20.2 Broken defences"]
N3["20.3 Poisoning and backdoors"]
N4["20.4 Privacy"]
N5["20.5 LLM applications"]
N6["20.6 Using ATLAS case studies"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX058"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["20.1 Adversarial examples"]
N2["20.2 Broken defences"]
N3["20.3 Poisoning and backdoors"]
N4["20.4 Privacy"]
N5["20.5 LLM applications"]
N6["20.6 Using ATLAS case studies"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX058"]
[Learning outcomes]
|
v
[20.1 Adversarial examples]
|
v
[20.2 Broken defences]
|
v
[20.3 Poisoning and backdoors]
|
v
[20.4 Privacy]
|
v
[20.5 LLM applications]
|
v
[20.6 Using ATLAS case studies]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX058]
Learning outcomes
You will (1) explain landmark results in adversarial ML and their lessons; (2) analyse case studies of attacks on LLM applications; (3) use MITRE ATLAS case studies to learn from real incidents; and (4) extract defensive lessons from each case.
20.1 Adversarial examples
Szegedy et al. (arXiv:1312.6199) found that imperceptible changes fool image classifiers; Goodfellow et al. (arXiv:1412.6572) explained it with local linearity. Lesson: high accuracy does not imply robustness.
The evasion lab repeats the finding on a toy scale: accuracy of 0.985 on clean points drops to 0.840 with a budget of one half. For DocSort, the equivalent question is how small a change to a letter's wording moves it to the wrong route. A model's test accuracy alone cannot answer that question, since test letters were never crafted.
20.2 Broken defences
Athalye et al. (arXiv:1802.00420) showed many published defences only obscured gradients; adaptive attacks broke them. Lesson: evaluate defences against attackers who know them.
For LCH this means a vendor claim that DocSort resists crafted letters is weak evidence unless the vendor's test assumed an attacker who knew the defence. Rina asks vendors which attacks were used and whether they were adaptive. A defence that only hides gradients may stop a lazy attack and fail against a careful one, which is the attacker that matters.
20.3 Poisoning and backdoors
Gu et al. BadNets (arXiv:1708.06733) showed backdoored models that behave normally until a trigger appears. Carlini et al. (arXiv:2302.10149) showed poisoning web-scale datasets is practical. Lesson: data and model provenance matter.
The poisoning lab shows the same pattern: 80 triggered points lift attack success to 0.980 while clean accuracy stays at 0.968, so normal testing sees nothing wrong. DocSort's monthly retraining on contractor labels is exactly the kind of pipeline where such a trigger could enter, carried in by a single careless or hostile labeller over several months.
20.4 Privacy
Shokri et al. (arXiv:1610.05820) introduced membership inference; Carlini et al. (arXiv:2012.07805) extracted memorised training text from a large language model. Lesson: models can leak training data.
Lab M reproduces the membership effect on a toy model. For LCH, the lesson applies to any model trained on patient records, including NoShow, and to any future language model fine-tuned on clinic letters, which could repeat fragments of them. Coarse outputs and minimised training data reduce that exposure, but do not remove it.
20.5 LLM applications
Greshake et al. (arXiv:2302.12173) demonstrated indirect prompt injection against LLM-integrated applications through retrieved content, leading to data theft and manipulation. Lesson: anything the model reads can instruct it; limit what the model can do.
Ask Lumen fits this pattern closely: it reads clinic documents and partner uploads, and it can call a booking tool. Rina's demonstration in the worked example below used the same mechanism, and the control that limited the damage was the booking tool's own checks, not the model's judgement about what text to obey.
20.6 Using ATLAS case studies
MITRE ATLAS documents case studies mapped to tactics and techniques. Read them to see how attacks combine reconnaissance, access and impact, and which mitigations would have helped.
Rina chose two cases for her briefing: one involving poisoned training data and one involving prompt injection through retrieved content. For each she listed which LCH system had a similar path and whether LCH already had the mitigation. One gap emerged: DocSort had no check on training-label provenance, which the next chapters address.
20.7 Worked example (synthetic LCH case)
Rina presented the indirect-injection case to LCH staff using a synthetic demonstration: a clinic leaflet uploaded by a partner contained hidden text instructing the chatbot to tell patients a clinic was closed. In the test system, Ask Lumen repeated the false closure. The lesson connected directly to the partner-upload review introduced earlier.
Staff responded more to the demonstration than to the papers, because they saw their own chatbot repeat the false message. Several staff asked how partner uploads were checked, which gave the review process the support it needed.
20.8 Errors and limits
- Learning attacks without their defensive lessons.
- Treating old cases as irrelevant to new models.
Each case study is useful only with its lesson attached; knowing that BadNets exists does not help unless DocSort's labels are then tracked. The older image cases still apply to newer models, because the underlying weakness, sensitivity to small crafted changes, has not gone away. Rina's briefing paired each case with a concrete action for one of LCH's systems, so staff left with work rather than only examples.
Exercises
- EX058 — Lessons. State the defensive lesson of Athalye et al. Rubric: 3 = evaluate with adaptive attacks; obscured gradients ≠ robustness; 2 = partial; 1 = vague; 0 = wrong.
- EX059 — Backdoor. Why are backdoors hard to detect with normal testing? Rubric: 3 = normal behaviour on clean inputs; trigger unknown; 2 = partial; 1 = vague; 0 = wrong.
- EX060 — Leaflet case. What impact category and two controls apply to the leaflet case? Rubric: 3 = integrity/misinformation; source review, provenance labelling, answer grounding checks; 2 = one control; 1 = vague; 0 = none.
Synthesis
Foundational cases show recurring patterns: accuracy is not robustness, defences need adaptive evaluation, data and models need provenance, models leak, and LLMs obey what they read.
Continue studying — chapter 20 of 40
← Chapter 19 (draft) · Course chapter index · Chapter 21 (draft) →