Chapter 13 — AI and machine-learning security: risk scenarios
Lesson map
Lesson 13 published. Map, example, EX037.
flowchart TD
N0["Learning outcomes"]
N1["13.1 Scenario format"]
N2["13.2 Likelihood and impact"]
N3["13.3 Prioritising"]
N4["13.4 Linking to controls and tests"]
N5["13.4b Sources of scenarios"]
N6["13.7 Reviewing scenarios over time"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX037"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["13.1 Scenario format"]
N2["13.2 Likelihood and impact"]
N3["13.3 Prioritising"]
N4["13.4 Linking to controls and tests"]
N5["13.4b Sources of scenarios"]
N6["13.7 Reviewing scenarios over time"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX037"]
[Learning outcomes]
|
v
[13.1 Scenario format]
|
v
[13.2 Likelihood and impact]
|
v
[13.3 Prioritising]
|
v
[13.4 Linking to controls and tests]
|
v
[13.4b Sources of scenarios]
|
v
[13.7 Reviewing scenarios over time]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX037]
Learning outcomes
You will (1) write concrete risk scenarios for AI systems; (2) estimate likelihood and impact; (3) prioritise scenarios; and (4) link each to controls and tests.
13.1 Scenario format
A scenario names: actor, entry point, technique, asset affected, effect, and existing controls. Example: "A fraudster (actor) posts a product review (entry) containing instructions (indirect prompt injection) that make Ask Brightwell (asset) start full refunds without returns (effect); current controls: none on the returns tool." Concrete scenarios are testable; vague ones ("AI could be hacked") are not. Jonas's six Brightwell scenarios follow this format, which is why each one could later be linked to a test on staging. Every Brightwell scenario also names its existing control, or none.
13.2 Likelihood and impact
Likelihood: how easy, how motivated attackers are, whether it has happened elsewhere (MITRE ATLAS case studies help). Impact: money, people affected, legal exposure, reputation. Use simple scales (1–5) and write reasons. For Ask Brightwell, refund fraud through an injected review scored likelihood 4 because writing a review needs only an account and the technique is published, and impact 4 because refunds cost money directly. Cross-customer lookup scored impact 4 for personal data. Writing these reasons lets others challenge the score. When the returns tool was added, the written reasons showed exactly which part of the refund-fraud score needed re-rating.
13.3 Prioritising
Rank by impact × likelihood, then adjust for detectability and reversibility: a harm you will notice and can undo quickly ranks lower than a silent, permanent one. ReturnGuard evasion by splitting returns ranked lower partly because it shows up in weekly reports and can be reversed by recovering the refunds. Refund fraud through injected reviews ranked higher because it could run quietly and the money is hard to get back once paid. Detectability and reversibility moved the ranking more than the raw scores did.
13.4 Linking to controls and tests
For each high-priority scenario, list the control that bounds it and the test that shows the control works. This turns the risk register into an engineering backlog. Brightwell's top scenario, refund fraud via injected review, links to the policy service as the control and to the held-out injection suite as the test. The cross-customer lookup links to the ownership check and its cross-user unit test. When a test fails, the scenario shows which risk has reopened. Each scenario-control-test link is a row in the risk register.
13.4b Sources of scenarios
Draw scenarios from OWASP LLM Top 10 entries, MITRE ATLAS techniques and case studies, past incidents in your organisation, and the data-flow diagram's taint paths. Ask people who operate the system what worries them: customer-service staff often know the abuse patterns before security does. Revisit scenarios after each incident and each major change. Brightwell's scenarios came from several of these: OWASP LLM01 for the injected review, the DFD's taint path for the returns tool, and customer-service staff who mentioned split returns as a familiar fraud trick.
13.5 Worked example (synthetic Brightwell case)
Jonas writes six scenarios. Top three: refund fraud via injected review (impact 4, likelihood 4); cross-customer order lookup (impact 4, likelihood 3, already fixed and tested); token-flood cost attack (impact 3, likelihood 3). Lower: ReturnGuard evasion by splitting returns (impact 2, likelihood 3) — detectable in weekly reports. The cross-customer lookup ranks high on impact but its test already passes, so attention goes to refund fraud. Splitting returns is lower partly because weekly reports would show it, which follows the rule of ranking silent harms above visible ones. Each score has a written reason in the register.
13.6 Errors and limits
- Vague scenarios.
- Scores without reasons.
- Scenarios with no linked test.
Jonas's six scenarios avoid these: each names an actor, an entry point and an effect, each score has a written reason, and the top three link to a control and a test. A scenario that cannot be tested cannot tell anyone when the risk has gone down. The quarterly review checks these three properties for every scenario still open. Each property is a single column in Brightwell's scenario register, so a gap is visible to anyone reading it.
13.7 Reviewing scenarios over time
Brightwell reviews its AI risk scenarios each quarter and after any incident. A scenario whose likelihood has changed — for example, because Ask Brightwell gained a new tool for issuing refunds — is re-rated, and its linked tests are updated in the same review. The same review checks that the held-out suite still contains cases for the refund path, and adds new ones if the tool's parameters changed. Results go to the refund-fraud scenario's owner, who decides whether the re-rating changes the release plan.
Exercises
- EX037 — Scenario. Write a full scenario for CodePilot suggesting a malicious package. Rubric: 3 = all six elements; 2 = four or five; 1 = two or three; 0 = none.
- EX038 — Scoring. Score your scenario with reasons. Rubric: 3 = both scores with reasons, adjustment; 2 = scores with reasons; 1 = scores only; 0 = none.
- EX039 — Link. Link the top Brightwell scenario to a control and a test. Rubric: 3 = specific control and test with pass criterion; 2 = without criterion; 1 = vague; 0 = none.
Synthesis
Concrete, scored scenarios linked to controls and tests turn AI risk into engineering work.
Continue studying — chapter 13 of 30
← Chapter 12 (draft) · Course chapter index · Chapter 14 (draft) →