Chapter 31 — AI and machine-learning security: advanced cases
Lesson map
Lesson 31 published. Map, example, EX091.
flowchart TD
N0["Learning outcomes"]
N1["31.1 Multi-step agent attacks"]
N2["31.2 Persistent backdoors"]
N3["31.3 Extraction of training data"]
N4["31.4 Universal jailbreaks"]
N5["31.5 Residual risk for agents"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX091"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["31.1 Multi-step agent attacks"]
N2["31.2 Persistent backdoors"]
N3["31.3 Extraction of training data"]
N4["31.4 Universal jailbreaks"]
N5["31.5 Residual risk for agents"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX091"]
[Learning outcomes]
|
v
[31.1 Multi-step agent attacks]
|
v
[31.2 Persistent backdoors]
|
v
[31.3 Extraction of training data]
|
v
[31.4 Universal jailbreaks]
|
v
[31.5 Residual risk for agents]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX091]
Learning outcomes
You will (1) analyse advanced attack scenarios: multi-step agent attacks, backdoors that survive safety training, training-data extraction and universal jailbreaks; (2) evaluate which defences address them; and (3) reason about residual risk for agentic systems.
31.1 Multi-step agent attacks
Agents chain tools. An injected document may instruct the agent to search for data with one tool and send it with another; each call may look harmless alone. Controls must consider sequences: taint tracking (marking data derived from untrusted inputs), restrictions on combining sensitive reads with outbound actions, and confirmation for outbound messages.
In Ask Lumen's planned e-mail feature, the sensitive read was the patient's history and the outbound action was the e-mail. Taint tracking would mark the history as sensitive and block it from leaving except to the patient's own address. Confirmation adds a human check on the final step.
31.2 Persistent backdoors
Hubinger et al. (arXiv:2401.05566) trained language models with hidden behaviours that persisted through standard safety training. Lesson: behavioural testing after training cannot prove absence of backdoors; provenance of models and training data matters.
For LCH, this argues for buying models from vendors who can show how training data was sourced and checked, and for keeping DocSort's own training labels traceable. Testing a model's behaviour before deployment remains useful, but a clean test result is not evidence that no trigger exists; it only shows that the tested inputs did not reveal one.
31.3 Extraction of training data
Carlini et al. (arXiv:2012.07805) extracted memorised sequences, including personal information, from a large language model. Fine-tuning on sensitive records can increase memorisation; deduplication and differential privacy reduce it.
If LCH ever fine-tunes a language model on clinic letters, these findings apply directly: repeated letter templates and rare patient details are the material most likely to be memorised. Deduplicating letters and removing names before training would be the first steps, followed by a test that tries to extract known synthetic records.
31.4 Universal jailbreaks
Zou et al. (arXiv:2307.15043) found automatically generated suffixes that cause aligned models to comply with harmful requests and transfer between models. Wei et al. (arXiv:2307.02483) explained jailbreak success through competing objectives and mismatched generalisation. Lesson: model-level refusals are a weak boundary; application controls must not depend on them.
For Ask Lumen, a successful jailbreak might make the model rude or off-topic, but it still cannot book for another patient, because the policy check does not rely on the model refusing.
31.5 Residual risk for agents
Even with strong controls, some injection attempts will succeed in influencing the model. Design so that the worst outcome of a successful injection is acceptable: limited tools, least privilege, confirmation and monitoring.
For Ask Lumen, Rina wrote down the worst outcome assuming injection succeeds every time: a wrong booking for the logged-in patient, which they must confirm, or a false statement about a clinic, which the grounding check catches against structured data. Both are acceptable with monitoring. An e-mail tool to arbitrary addresses would not be, which led to the restriction in the worked example.
31.6 Worked example (synthetic LCH case)
LCH considered giving Ask Lumen an e-mail tool to send appointment summaries. Rina's analysis showed a two-step path: an injected document could make the agent read another patient's appointment (blocked by delegated tokens) or the current patient's full history (allowed) and e-mail it to an attacker's address. LCH restricted the e-mail tool to the patient's registered address only, with a fixed template.
The fixed template also stops an injected document from adding its own text to the e-mail. Together with the fixed recipient, the worst case becomes a patient receiving their own data.
31.7 Errors and limits
- Analysing tool calls one at a time.
- Trusting safety training as a security boundary.
- Ignoring memorisation in fine-tuned models.
Each call in the e-mail attack looked legitimate alone: reading one's own history and sending a summary are both normal. Only the sequence was harmful. Safety training reduces harmful answers but does not stop determined attackers, as the jailbreak research shows. Fine-tuned models trained on patient letters may repeat them, so memorisation must be tested before deployment and after each retraining.
Exercises
- EX091 — Chain. Describe a two-tool exfiltration chain and two controls. Rubric: 3 = read + send chain; registered-address restriction, taint-aware confirmation; 2 = one control; 1 = vague; 0 = none.
- EX092 — Backdoor lesson. What does the Sleeper Agents result imply for using third-party models? Rubric: 3 = testing cannot prove absence; provenance and trusted sources matter; 2 = partial; 1 = vague; 0 = wrong.
- EX093 — Jailbreak lesson. Why should application controls not depend on model refusals? Rubric: 3 = refusals bypassable (universal/transfer attacks); enforce externally; 2 = partial; 1 = vague; 0 = wrong.
Synthesis
Advanced cases show attacks that chain, persist and transfer. The design response is to bound the worst outcome of a successful manipulation rather than hope it never happens.
Continue studying — chapter 31 of 40
← Chapter 30 (draft) · Course chapter index · Chapter 32 (draft) →