Chapter 26 — AI and machine-learning security: methods II
Lesson map
Lesson 26 published. Map, example, EX076.
flowchart TD
N0["Learning outcomes"]
N1["26.1 Red-teaming LLM applications"]
N2["26.2 Test suites"]
N3["26.3 Adaptive attacks"]
N4["26.4 Coverage and reporting"]
N5["26.5 Automation"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX076"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["26.1 Red-teaming LLM applications"]
N2["26.2 Test suites"]
N3["26.3 Adaptive attacks"]
N4["26.4 Coverage and reporting"]
N5["26.5 Automation"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX076"]
[Learning outcomes]
|
v
[26.1 Red-teaming LLM applications]
|
v
[26.2 Test suites]
|
v
[26.3 Adaptive attacks]
|
v
[26.4 Coverage and reporting]
|
v
[26.5 Automation]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX076]
Learning outcomes
You will (1) plan adversarial evaluations of LLM applications (red-teaming) safely; (2) build test suites for injection, disclosure and excessive agency; (3) use adaptive attacks against your own defences; and (4) report results with intervals and coverage.
26.1 Red-teaming LLM applications
Red-teaming is structured adversarial testing: a team tries to achieve defined attacker goals within an authorised scope. NIST AI 600-1 lists red-teaming among suggested actions for generative AI risk. Define goals (e.g., make the chatbot reveal another patient's appointment; make it book without consent), rules of engagement, and reporting.
For Ask Lumen, the rules of engagement named the staging copy as the only target, limited testing to two weeks, required synthetic patients, and named LCH's security lead as the contact if anything unexpected happened. The report format was agreed in advance, so findings could not be softened afterwards.
26.2 Test suites
Turn goals into suites: many test cases varying wording, language, position of injected text (start, middle, end of a document), encoding and multi-turn strategies. Keep a held-out portion that developers never see. Use harmless payloads and synthetic data.
Position matters because a model may treat an instruction at the end of a long leaflet differently from one in the first line. Rina's suite placed each payload in all three positions and in a table, and wrote some multi-turn cases where the patient's second message referred back to the document. The held-out portion was written by a classmate and stored separately.
26.3 Adaptive attacks
After a defence is added, attack the defence: if documents are wrapped in delimiters, try text that imitates the delimiters; if a classifier flags "ignore previous", try paraphrases. Carlini et al. (arXiv:1902.06705) stress adaptive evaluation; it applies equally to LLM defences.
When Ask Lumen began wrapping partner documents in labelled tags, Rina's next round included text that closed the tag early and opened a fake "system" block. Some attempts changed the model's answer, but none produced a booking, because the policy check did not depend on the tags.
26.4 Coverage and reporting
Report per-category success rates with intervals, the total number of attempts and the categories not tested. Coverage maps to OWASP LLM entries help readers see gaps.
Rina's report on 100 false-information cases gave 9 successes with a Wilson interval of about 4.8% to 16.2%, and stated plainly that attacks in Portuguese and attacks hidden in images had not been tested. The coverage table showed LLM01, LLM02, LLM06 and LLM10 covered, and LLM03 left to the supply-chain review, with a named owner and a date for testing.
26.5 Automation
Scripts can generate variants from templates, run them, and grade outcomes from tool logs rather than from the model's text. Validate any automatic grader against human labels on a sample.
For cross-patient booking, Rina's grader read the scheduling sandbox's log and counted bookings whose patient differed from the session's patient. That is exact. For false information, grading needed judgement, so she checked the automatic grader against her own labels on 60 transcripts and found two disagreements, which led her to tighten its rule.
26.6 Worked example (synthetic LCH case)
Rina's authorised red-team exercise on Ask Lumen staging used 600 cases in six categories. Results: cross-patient booking 0/100; disclosure of other patients' data 2/100 (both through a summarisation of a staff note left in the index); false clinic information from injected documents 9/100; cost exhaustion blocked by quotas. The two disclosures led to removing staff notes from the index; the false-information rate led to a grounding check against structured clinic data.
The staff-note disclosures came from content, not from the model's rules: removing the notes fixed the cause.
26.7 Errors and limits
- Testing only static, public jailbreak lists.
- Grading by the model's own words.
- Unscoped "red-teaming" of production.
Public jailbreak lists are what Ask Lumen's vendor has probably already seen, so passing them shows little. A grader that reads the model's answer, such as "I have booked that", misses bookings made silently and counts boasts that did nothing; tool logs show what really happened. Testing production without scope risks real patients' appointments, which is why Rina tested only staging under written approval.
Exercises
- EX076 — Goals. Write three red-team goals for DocSort. Rubric: 3 = three specific, measurable goals; 2 = two; 1 = vague; 0 = none.
- EX077 — Adaptive. Propose an adaptive test for a delimiter-based defence (described safely). Rubric: 3 = imitate/close delimiters to make content look like instructions; 2 = generic; 1 = vague; 0 = harmful detail.
- EX078 — Grading. Why grade from tool logs rather than responses? Rubric: 3 = actions may happen despite refusing text; logs show real effects; 2 = partial; 1 = vague; 0 = wrong.
Synthesis
Methods II builds disciplined adversarial testing: scoped red-teaming, held-out suites, adaptive attacks and log-based grading reported with intervals and coverage.
Continue studying — chapter 26 of 40
← Chapter 25 (draft) · Course chapter index · Chapter 27 (draft) →