Chapter 4 — AI and machine-learning security: core methods
Lesson map
Lesson 04 published. Map, example, EX016.
flowchart TD
N0["Learning outcomes"]
N1["4.1 Attack classes"]
N2["4.2 Testing for injection"]
N3["4.3 Testing classifiers"]
N4["4.4 Privacy attacks"]
N5["4.5 Matching methods to risks"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX016"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["4.1 Attack classes"]
N2["4.2 Testing for injection"]
N3["4.3 Testing classifiers"]
N4["4.4 Privacy attacks"]
N5["4.5 Matching methods to risks"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX016"]
[Learning outcomes]
|
v
[4.1 Attack classes]
|
v
[4.2 Testing for injection]
|
v
[4.3 Testing classifiers]
|
v
[4.4 Privacy attacks]
|
v
[4.5 Matching methods to risks]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX016]
Learning outcomes
You will (1) describe the main attack classes and how each is tested; (2) test LLM applications for direct and indirect prompt injection; (3) test classifiers for evasion; (4) understand privacy attacks; and (5) choose methods to match risks.
4.1 Attack classes
Evasion: inputs crafted to fool a model (Goodfellow et al., arXiv:1412.6572). Poisoning: corrupting training data or adding backdoors (Gu et al., arXiv:1708.06733). Privacy: membership inference (Shokri et al., arXiv:1610.05820) and model extraction (Tramèr et al., arXiv:1609.02943). Prompt injection: direct (typed by the user) and indirect (hidden in content the model reads; Greshake et al., arXiv:2302.12173).
At Meridian, prompt injection applies to ClaimChat, evasion to FraudFlag, and membership inference to any model trained on claimant data. Map each class to the services where it applies before writing a test plan.
4.2 Testing for injection
Build test inputs that try to make the assistant do something it should not: book a callback for another claim, reveal its instructions, mark a claim as approved, include a link carrying data. Plant them in messages and in documents. Vary wording, language and placement. Judge by effects in tool logs and data, not by the tone of replies.
Sofia's 60 indirect cases were planted in uploaded quotes and text in photos. Each case had a harmless marker, so success showed in the tool log rather than in a judgement about wording. The direct cases came from the chat window.
4.3 Testing classifiers
For FraudFlag, change the features a fraudster controls (claim amount split, timing, description) and measure how often known fraud cases fall below the flag threshold.
"""Lab E: evasion of a toy linear classifier (synthetic data, stdlib only).
Trains logistic regression on two synthetic clusters, then applies a
fast-gradient-sign (FGSM-style) perturbation of size eps to every test point
and reports clean vs adversarial accuracy. Teaching model only."""
import math, random
rng = random.Random(7)
def point(label):
cx = 1.0 if label else -1.0
return [rng.gauss(cx, 0.7), rng.gauss(cx, 0.7)], label
train = [point(i % 2) for i in range(400)]
test = [point(i % 2) for i in range(200)]
w, b = [0.0, 0.0], 0.0
sig = lambda z: 1 / (1 + math.exp(-z))
for epoch in range(200):
gw, gb = [0.0, 0.0], 0.0
for x, y in train:
p = sig(w[0]*x[0] + w[1]*x[1] + b)
gw[0] += (p - y)*x[0]; gw[1] += (p - y)*x[1]; gb += p - y
w = [w[i] - 0.1*gw[i]/len(train) for i in range(2)]; b -= 0.1*gb/len(train)
def acc(data, eps):
ok = 0
for x, y in data:
# gradient of the loss w.r.t. x is (p - y) * w; FGSM steps along its sign
p = sig(w[0]*x[0] + w[1]*x[1] + b)
xa = [x[i] + eps*math.copysign(1, (p - y)*w[i]) for i in range(2)]
ok += (sig(w[0]*xa[0] + w[1]*xa[1] + b) >= 0.5) == bool(y)
return ok / len(data)
print(f"weights w=({w[0]:.3f}, {w[1]:.3f}) b={b:.3f}")
for eps in (0.0, 0.25, 0.5, 0.75, 1.0):
print(f"eps={eps:.2f} accuracy={acc(test, eps):.3f}")
Expected output:
weights w=(1.795, 1.756) b=-0.080
eps=0.00 accuracy=0.985
eps=0.25 accuracy=0.950
eps=0.50 accuracy=0.840
eps=0.75 accuracy=0.660
eps=1.00 accuracy=0.505
Measure the share of known fraud cases that drop below the threshold after each change, and report it with an interval. For FraudFlag, Sofia used 30 variations on known synthetic fraud claims. Test only the features a fraudster can actually control, since other features are not available to them. Keep the variations realistic, such as amounts and wording.
4.4 Privacy attacks
Ask whether outputs reveal training data or allow copying the model. Detailed scores and unlimited queries increase both risks.
For FraudFlag, a partner API that returns detailed scores without limits would make both easier. Membership itself can be sensitive: knowing that someone made a claim can reveal personal circumstances. Sofia's lab recommendation was to return only "standard / review" with per-partner rate limits. Test with synthetic claimant records in staging. Measure how much better than chance an attacker could guess membership, and how many queries it would take to copy the model's behaviour, before and after coarse outputs.
4.5 Matching methods to risks
Start from the risks in your data-flow picture. Uploaded documents → indirect injection tests. Tools → agency tests. Scoring API → extraction and evasion tests. Personal data in training → membership tests.
Sofia's plan for ClaimChat followed this mapping: indirect injection tests for uploads, agency tests for the callback tool and disclosure tests for claim status. Keep the mapping with the plan. Each pairing follows from an entry point or asset in the picture. If the picture shows a route with no tests mapped to it, either add tests or record why the route is out of scope, so the gap is deliberate and visible to the reviewer and owner.
4.6 Worked example (synthetic Meridian case)
Sofia's test plan for ClaimChat: 40 direct and 60 indirect injection cases (in uploaded quotes and photos' text), 20 disclosure cases, 10 consumption cases. For FraudFlag: 30 evasion variations on known synthetic fraud claims. In the evasion analogue, the lab shows a toy model's accuracy dropping from 0.985 to 0.505 as perturbations grow — a reason not to let one score decide a claim.
Each set of cases traces to a risk in the data-flow picture, so a reviewer can see why it is there. The plan was approved before any case was run.
4.7 Errors and limits
- Judging tests by reply tone.
- Only direct injection tests.
- Methods chosen without reference to risks.
Each error misses real harm. A polite refusal can hide a tool call that still happened, so tone is not evidence. Direct tests alone miss injection through uploaded quotes and photos, ClaimChat's main exposure. Methods chosen without reference to risks waste effort and leave mapped risks untested. Check your test plan for all three before running it. Fix each before running cases. Sofia's own plan avoided them by mapping cases to the data-flow picture.
Exercises
- EX016 — Classes. Match four scenarios to attack classes. Rubric: 3 = four correct; 2 = three; 1 = one or two; 0 = none.
- EX017 — Indirect cases. Describe three indirect-injection test documents (not weaponised) with expected safe behaviour. Rubric: 3 = three varied with expectations; 2 = two; 1 = one; 0 = none.
- EX018 — Evasion. Design two evasion tests for FraudFlag. Rubric: 3 = two attacker-controlled changes with measurement; 2 = one; 1 = vague; 0 = none.
- EX019 — Lab. What does the drop to 0.505 mean for FraudFlag decisions? Rubric: 3 = crafted input can flip decisions; keep human review; toy caveat; 2 = partial; 1 = vague; 0 = wrong.
- EX020 — Matching. Match three ClaimChat risks to test methods. Rubric: 3 = three apt; 2 = two; 1 = one; 0 = none.
Synthesis
Core methods test each attack class where the system is exposed, judged by effects, chosen to match the risks you have mapped.
Continue studying — chapter 04 of 12
← Chapter 03 (draft) · Course chapter index · Chapter 05 (draft) →