Chapter 23 — AI and machine-learning security: methods I
Lesson map
Lesson 23 published. Map, example, EX067.
flowchart TD
N0["Learning outcomes"]
N1["23.1 Threat analysis"]
N2["23.2 Mapping"]
N3["23.3 Prioritising"]
N4["23.4 Selecting tests"]
N5["23.4b Attack trees"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX067"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["23.1 Threat analysis"]
N2["23.2 Mapping"]
N3["23.3 Prioritising"]
N4["23.4 Selecting tests"]
N5["23.4b Attack trees"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX067"]
[Learning outcomes]
|
v
[23.1 Threat analysis]
|
v
[23.2 Mapping]
|
v
[23.3 Prioritising]
|
v
[23.4 Selecting tests]
|
v
[23.4b Attack trees]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX067]
Learning outcomes
You will (1) perform a structured threat analysis for an AI system; (2) map threats to OWASP LLM 2025 and MITRE ATLAS; (3) prioritise threats by likelihood and impact; and (4) select tests for top threats.
23.1 Threat analysis
For each component and flow in the system model, ask what an attacker could do: inject instructions, poison data, evade detection, extract the model or data, exhaust resources, or abuse tools. Include insiders and accidents with similar effects.
For Ask Lumen's document store, Rina asked each question in turn. Inject instructions: yes, through partner uploads. Poison: not applicable, since no training happens there. Extract data: possibly, if the chatbot quotes internal documents. Exhaust resources: long conversations. An accident with similar effect was a clinic uploading an outdated leaflet that listed wrong hours.
23.2 Mapping
Map threats to OWASP LLM 2025 entries (for example LLM01 prompt injection, LLM02 sensitive information disclosure, LLM06 excessive agency, LLM10 unbounded consumption) and to MITRE ATLAS tactics and techniques. Mapping helps check coverage and find documented mitigations.
Rina's mapping placed partner-document injection under LLM01, cross-patient booking under LLM06 and long-chat cost under LLM10. Checking the list for gaps, she noticed she had no entry for LLM02 on the booking tool's responses, and added a test that the chatbot never repeats another patient's appointment details.
23.3 Prioritising
Rate likelihood (attacker access, skill, motivation) and impact (patient harm, privacy, cost, reputation). A simple 1–3 scale and a written justification are enough. Revisit when things change.
The justification matters more than the number. For cross-patient booking, Rina wrote that likelihood was 2 because delegated tokens now block the direct path but the patient's own carers can still be misused; impact was 3 because a wrong booking can delay care. Another reviewer can then challenge the reasoning rather than the score itself, and update it when tokens or carer rules change.
23.4 Selecting tests
For each top threat, choose a test: an injection suite, a leakage suite, a poisoning simulation, a load test. Define pass criteria in advance.
For partner-document injection, the pass criterion was set before testing: no more than one false clinic statement in 200 injected documents, and zero bookings caused by document text. Setting the threshold first stops the team from deciding afterwards that whatever result appeared was acceptable. A test without a criterion only describes behaviour; it cannot fail, and so cannot show that a control works.
23.4b Attack trees
An attack tree breaks an attacker goal into sub-goals joined by AND or OR. For "book an appointment as another patient" the OR branches might be: steal the patient's credentials; manipulate the chatbot into booking for them; exploit the booking API directly. Each branch can be estimated and controlled separately, and the tree shows which single control blocks several branches — for example, a policy check tied to the logged-in patient blocks both the chatbot and some API paths.
For LCH, the credential branch is handled by the portal's sign-in controls, outside the AI team's scope.
23.5 Worked example (synthetic LCH case)
Rina's top three for Ask Lumen: (1) injection through partner documents causing false information (likelihood 2, impact 3); (2) booking for another patient through manipulation (2, 3; mitigated by delegated tokens, retest needed); (3) cost exhaustion through long chats (3, 2). Tests: 200 injected documents measuring false statements; 400 cross-patient attempts; a load test with token quotas.
Cost exhaustion ranked highest on likelihood because it needs no skill, only patience, and the quotas test was cheap to run. Cross-patient booking needed a retest because the delegated-token change was recent and had not been tested under manipulation.
23.6 Errors and limits
- Threat lists not linked to the system model.
- No justification for ratings.
- Tests chosen without pass criteria.
Threats listed without reference to the system model tend to be generic, such as "model may be hacked", and point to no control. Ratings without reasons cannot be reviewed or updated when conditions change. Tests without pass criteria produce results that can be read either way, so nobody knows whether a control worked. Rina's threat register links each threat to a diagram flow, a reason and a criterion.
Exercises
- EX067 — Threat list. List five threats for NoShow with OWASP or ATLAS mapping where relevant. Rubric: 3 = five plausible with mappings; 2 = four; 1 = two; 0 = none.
- EX068 — Prioritise. Rate one threat's likelihood and impact with justification. Rubric: 3 = both rated with reasons; 2 = no reasons; 1 = vague; 0 = none.
- EX069 — Test design. Design the test for threat (2) with pass criterion. Rubric: 3 = suite size, method, metric, threshold; 2 = three; 1 = vague; 0 = none.
Synthesis
Methods I turns the system model into a prioritised threat list mapped to shared catalogues, each with a test and a pass criterion.
Continue studying — chapter 23 of 40
← Chapter 22 (draft) · Course chapter index · Chapter 24 (draft) →