Chapter 23 — AI and machine-learning security: threat and failure analysis
Lesson map
Lesson 23 published. Map, example, EX067.
flowchart TD
N0["Learning outcomes"]
N1["23.1 Structured threat analysis"]
N2["23.2 Mapping to shared catalogues"]
N3["23.3 Combining with failure analysis"]
N4["23.4 Ranking"]
N5["23.4b Keeping the analysis alive"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX067"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["23.1 Structured threat analysis"]
N2["23.2 Mapping to shared catalogues"]
N3["23.3 Combining with failure analysis"]
N4["23.4 Ranking"]
N5["23.4b Keeping the analysis alive"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX067"]
[Learning outcomes]
|
v
[23.1 Structured threat analysis]
|
v
[23.2 Mapping to shared catalogues]
|
v
[23.3 Combining with failure analysis]
|
v
[23.4 Ranking]
|
v
[23.4b Keeping the analysis alive]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX067]
Learning outcomes
You will (1) run a structured threat analysis on an AI system using STRIDE-style prompts adapted to AI; (2) map threats to MITRE ATLAS techniques and OWASP LLM entries; (3) combine threat and failure analysis; and (4) rank threats by risk.
23.1 Structured threat analysis
Walk each element of the data-flow diagram and ask: can it be spoofed (fake sensor, fake supplier), tampered (poisoned data, modified weights), repudiated (untraceable agent action), disclosed (membership inference, prompt leakage), denied (resource exhaustion), or used to elevate privilege (prompt injection reaching a tool)? Add AI-specific prompts: can the model be evaded, extracted, backdoored, or made to confabulate convincingly?
Walking element by element keeps the analysis complete in a way that brainstorming does not. Each question is asked even when the answer seems obvious, and 'not applicable' is written down with a reason.
23.2 Mapping to shared catalogues
MITRE ATLAS organises adversary behaviour against AI into tactics (such as reconnaissance, resource development, initial access, ML model access, persistence, defense evasion, discovery, collection, ML attack staging, exfiltration and impact) and techniques, with real-world case studies. Mapping your threats to ATLAS techniques and OWASP LLM entries helps reviewers check coverage and links to known mitigations.
Mapping is a cross-check rather than a substitute for analysis. If a catalogue technique has no matching threat in the model, the team asks whether it was missed or genuinely does not apply. Case studies are especially useful for convincing stakeholders that a threat is real, since they describe attacks that actually happened.
23.3 Combining with failure analysis
Threats and failures often share effects: a missed defect can come from glare (failure) or a marking (attack). Analyse them together, so that controls such as stratified monitoring and manual fallback cover both.
Joint analysis also avoids duplicated work and conflicting controls. A manual fallback designed for glare can be triggered by an attacker who wants to slow inspections, so the fallback itself needs a rate limit or an alert. Sam's hazard and threat tables share identifiers for effects, so a reviewer can see every cause, accidental or deliberate, that leads to a missed defect, and check that some control covers each of them.
23.4 Ranking
Estimate likelihood (attacker capability and motivation, exposure) and impact (safety, financial, legal, reputational). Use a simple ordinal scale and document reasoning. Re-rank when the system or threat landscape changes.
Ordinal scales such as low, medium and high are honest about precision: likelihood estimates for attacks are judgements, not measurements. Writing the reasoning beside each rating, for example 'medium: supplier bulletins are public and editable by any supplier staff', lets reviewers challenge the judgement rather than the number. Re-ranking is triggered by events, such as a new tool or a published attack, not only by the calendar.
23.4b Keeping the analysis alive
Threat models age. New tools, data sources, model versions and published attacks change the picture. Store the analysis next to the architecture, give it a version, and schedule review at each significant change and at least yearly. Record which threats were accepted and by whom, so that accepted risks are not forgotten.
For Wrench, the trigger is concrete: when Bulletin Gate went into service, Sam's threat 1, injection via supplier bulletin, had to be re-scored, because bulletins now reach Wrench only as structured records. The likelihood fell, but a new threat appeared in its place: a poisoned update to the extraction model, which the Chapter 47 trial later tested.
23.5 Worked example (synthetic NRS case)
Sam's top five for Wrench: (1) indirect injection via supplier bulletin → work-order tool (likelihood medium, impact high); (2) contractor-note injection (medium/high); (3) sensitive disclosure from HR documents in retrieval (medium/medium, mitigated in Chapter 13); (4) system-prompt extraction revealing tool names (high/low); (5) resource exhaustion via very long documents (medium/medium). Each was mapped to OWASP (LLM01, LLM01, LLM02, LLM07, LLM10) and to ATLAS techniques for prompt injection and denial of ML service.
The two injection threats share an entry in each catalogue but stay separate in the list because their sources, and thus their controls, differ.
23.6 Errors and limits
- Threat lists without the system model behind them.
- Ignoring non-adversarial failures with the same effect.
- Rankings never revisited.
A threat list without a system model is generic and misses the paths specific to the design. Ignoring failures that have the same effect as attacks leads to duplicate controls or gaps. Rankings frozen at design time drift out of date as tools and data sources change. Sam ties the threat list to the diagram's version, so any change to the diagram flags the analysis for review.
Exercises
- EX067 — STRIDE for AI. Apply the six STRIDE prompts to Pulse, one threat each. Rubric: 3 = six plausible; 2 = four; 1 = two; 0 = none.
- EX068 — Mapping. Map "attacker buys an expired domain referenced in a training URL list" to an ATLAS tactic and OWASP entry. Rubric: 3 = resource development/initial access or poisoning; LLM04; 2 = one; 1 = vague; 0 = wrong.
- EX069 — Ranking. Rank Sam's five threats and justify the top one. Rubric: 3 = sensible ranking with likelihood and impact reasoning; 2 = ranking without reasons; 1 = vague; 0 = none.
Synthesis
Threat analysis turns the system model into a ranked list of what could go wrong, mapped to shared catalogues and combined with failure analysis so controls cover both.
Continue studying — chapter 23 of 50
← Chapter 22 (draft) · Course chapter index · Chapter 24 (draft) →