Chapter 8 — AI and machine-learning security: difficult cases
Lesson map
Lesson 08 published. Map, example, EX036.
flowchart TD
N0["Learning outcomes"]
N1["8.1 Conflicting controls"]
N2["8.2 Multi-agent and supply chain"]
N3["8.3 Privacy leakage"]
N4["8.4 Incidents with uncertain cause"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX036"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["8.1 Conflicting controls"]
N2["8.2 Multi-agent and supply chain"]
N3["8.3 Privacy leakage"]
N4["8.4 Incidents with uncertain cause"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX036"]
[Learning outcomes]
|
v
[8.1 Conflicting controls]
|
v
[8.2 Multi-agent and supply chain]
|
v
[8.3 Privacy leakage]
|
v
[8.4 Incidents with uncertain cause]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX036]
Learning outcomes
You will (1) analyse cases where controls conflict or evidence is ambiguous; (2) handle multi-agent and supply-chain cases; (3) reason about privacy leakage from fine-tuned models; (4) manage incidents with uncertain root cause; and (5) document judgment under ambiguity.
8.1 Conflicting controls
Logging every prompt supports investigation but stores personal data; strict refusal reduces misuse but harms legitimate users; human confirmation bounds harm but causes approval fatigue. Difficult cases are rarely solved by one more control; they need explicit trade-offs, owners and monitoring of both sides.
At Corvane, the depot incident showed one such trade-off: automatic rebooking saves time but trusted email text too much. Write each trade-off down with its owner and the measure that shows whether the balance is right.
8.2 Multi-agent and supply chain
When agents call agents, an injection in one can propagate. Treat every agent-to-agent message as untrusted input and keep enforcement at each tool boundary. Supply-chain cases include poisoned public datasets — Carlini et al. (arXiv:2302.10149) show that poisoning web-scale datasets is practical — and model files from public hubs. Pin hashes, record provenance and prefer formats that do not execute code on load.
"""Lab H: integrity of model artefacts with SHA-256 manifests (stdlib only).
Creates synthetic 'artefact' bytes, records a manifest, then detects a
one-byte tampering and a swapped file. No real models are involved."""
import hashlib, json
files = {"model.bin": bytes(range(256)) * 40, "tokenizer.json": b'{"vocab": ["a", "b"]}',
"config.yaml": b"layers: 4\nheads: 8\n"}
manifest = {n: hashlib.sha256(b).hexdigest() for n, b in files.items()}
print(json.dumps({k: v[:16] for k, v in manifest.items()}, indent=1))
received = dict(files)
tampered = bytearray(received["model.bin"]); tampered[5000] ^= 1
received["model.bin"] = bytes(tampered)
received["config.yaml"] = b"layers: 4\nheads: 8\ntrust_remote_code: true\n"
for n, b in sorted(received.items()):
ok = hashlib.sha256(b).hexdigest() == manifest[n]
print(f"{n:15s} {'OK' if ok else 'MISMATCH'}")
Expected output:
{
"model.bin": "e96760a87768717b",
"tokenizer.json": "f63d2ddabc61f2d9",
"config.yaml": "e5bbb570ca0fcb87"
}
config.yaml MISMATCH
model.bin MISMATCH
tokenizer.json OK
At Corvane, Dispatch Copilot's reader model passes fields to the planner. Lab H shows the check: a changed byte and an added configuration line are both detected.
8.3 Privacy leakage
Fine-tuned models can memorise and reveal training data; Carlini et al. (arXiv:2012.07805) extracted verbatim training examples from a language model. When a model is fine-tuned on customer messages, test for verbatim regurgitation and membership inference, and consider deduplication, filtering and differential privacy.
At Corvane, Tomás's applied project tested this for a reply-suggestion model fine-tuned on customer messages, and deduplication cut membership advantage from 0.17 to 0.06. Run such tests after each fine-tune, not only once. Report leakage with the attack used and its conditions.
8.4 Incidents with uncertain cause
Was the wrong refund an injection, a model error, or a policy bug? Preserve prompts, retrieved documents, tool calls and versions; reproduce on an isolated copy; consider all three until evidence separates them. Contain first: lower limits, disable the tool, roll back the prompt.
Amira's depot incident followed this order: containment by disabling automatic rebooking, evidence from logs, then cause. The cause turned out to be a compromised carrier account, not a model error, which only the preserved evidence could show.
8.5 Worked example (synthetic Corvane case)
Copilot rebooked 14 shipments to a new depot after reading a carrier email. Was it an attack? The email said "Per updated routing, all Thursday loads go to Depot 7". Logs show the email came from a real carrier account that had been compromised. Amira contains by disabling automatic rebooking, notifies the carrier, and adds a rule: depot changes require a matching entry in the routing system, not just email text. She records that the model behaved as designed — the design trusted email too much.
8.6 Errors and limits
- Seeking a single control for a trade-off.
- Trusting agent-to-agent messages.
- Assigning blame to "the model" before evidence.
Each error leads to a wrong response. A trade-off cannot be solved by one more control; it needs an owner and monitoring of both sides. Messages from other agents can carry injected instructions, so enforcement is needed at each tool boundary. Blaming the model before evidence would have hidden the compromised carrier account behind Corvane's depot incident. Check each incident review for all three.
Exercises
- EX036 — Trade-off. Resolve the logging versus privacy conflict for Copilot. Rubric: 3 = retention, redaction, access, purpose limit; 2 = three; 1 = one; 0 = none.
- EX037 — Hash lab. What do the lab's MISMATCH lines tell an incident responder? Rubric: 3 = files changed since pinning; investigate provenance; block load; 2 = partial; 1 = vague; 0 = wrong.
- EX038 — Memorisation test. Design a test for verbatim leakage from a model fine-tuned on customer messages. Rubric: 3 = canaries or known records, prompts, measurement, threshold; 2 = three; 1 = vague; 0 = none.
- EX039 — Incident. List the evidence to preserve for the depot incident. Rubric: 3 = email, retrieval, prompts, tool calls, versions, timeline; 2 = four; 1 = two; 0 = none.
- EX040 — Root cause. Write the root-cause statement for §8.5. Rubric: 3 = design trust in email, compromised carrier, fix; 2 = two; 1 = blames model; 0 = none.
Synthesis
Difficult cases need explicit trade-offs, distrust of every message crossing an agent or supply-chain boundary, and incident handling that preserves evidence before deciding cause.
Continue studying — chapter 08 of 12
← Chapter 07 (draft) · Course chapter index · Chapter 09 (draft) →