Chapter 1 — Advanced security decision-making
Lesson map
Lesson 01 published. Map, example, EX001.
flowchart TD
N0["Learning outcomes"]
N1["1.1 Why AI-security decisions are different"]
N2["1.2 A decision record"]
N3["1.3 Structuring with the AI RMF"]
N4["1.4 Deciding under uncertainty"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX001"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["1.1 Why AI-security decisions are different"]
N2["1.2 A decision record"]
N3["1.3 Structuring with the AI RMF"]
N4["1.4 Deciding under uncertainty"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX001"]
[Learning outcomes]
|
v
[1.1 Why AI-security decisions are different]
|
v
[1.2 A decision record]
|
v
[1.3 Structuring with the AI RMF]
|
v
[1.4 Deciding under uncertainty]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX001]
Learning outcomes
You will (1) frame an AI-security decision as options, consequences and uncertainty; (2) separate facts, estimates and values in a decision record; (3) use the NIST AI RMF functions to structure organization-scale choices; (4) decide under incomplete evidence with explicit triggers for revisiting; and (5) communicate a decision so others can challenge it.
1.1 Why AI-security decisions are different
Most security decisions concern systems whose behaviour is specified by code. AI systems add behaviour learned from data and, for language models, behaviour steered by text that attackers can supply. Three consequences follow. First, failure rates are statistical: a control that blocks 97% of tested injections does not block "injection". Second, evidence ages quickly, because models, prompts and attacker techniques change. Third, the same model can be safe in one deployment and dangerous in another, depending on what tools and data it can reach. A senior specialist's job is therefore less to find a perfect control than to choose, with stated uncertainty, between imperfect options, and to arrange for the choice to be revisited.
1.2 A decision record
A useful record has seven parts: the question; the options (always including "do nothing" and "do not deploy"); the consequences of each for affected parties; the evidence, labelled as fact, estimate or assumption; the values and risk appetite applied; the decision with its owner; and the revisit triggers (a date, a metric threshold, a model change, an incident). Writing values separately matters: two people can agree on all facts and still disagree on whether a 1-in-1,000 wrong refund is acceptable. Hiding that disagreement inside a "risk score" prevents honest debate.
1.3 Structuring with the AI RMF
NIST AI 100-1 organises AI risk work into Govern, Map, Measure and Manage. Use it as a checklist for the decision: is there an accountable owner and policy (Govern)? Is the context — users, data, tools, affected people — mapped (Map)? Is there evidence about the relevant risks, with known limits (Measure)? Are treatments chosen, prioritised and monitored (Manage)? Gaps in any function signal where the decision is weakest. The Generative AI Profile (NIST AI 600-1) adds risks specific to generative systems, such as information integrity and information security, that should appear in the Map step for language-model deployments.
1.4 Deciding under uncertainty
You rarely have the evidence you want. Three tools help. Bounding: if zero failures were seen in n independent trials, the 95% upper bound on the failure rate is roughly 3/n, so 0 in 300 still allows about 1%. Reversibility: prefer options that can be undone cheaply — a feature flag, a refund limit, a staged rollout. Value of information: ask whether more testing could change the decision; if not, decide now.
Amira's refund decision uses all three: the 0-in-400 bound, a low cap that can be raised later, and a trigger instead of more testing.
1.5 Worked example (synthetic Corvane case)
The board asks Amira whether Dispatch Copilot may issue refunds automatically. Options: (a) no refunds by the agent; (b) refunds up to €50 with policy checks outside the model; (c) refunds up to €500. Evidence: in an isolated test, 0 of 400 adversarial conversations produced an out-of-policy refund under option (b) (upper bound ≈ 0.75%), while customer-service data estimate €180,000 a year in handling savings. Values: the finance director accepts up to €10,000 a year in wrongful refunds. Amira computes the worst plausible exposure at 0.75% of 60,000 refunds × €50 ≈ €22,500, above appetite; at a €20 limit the same bound gives €9,000, within it, so she recommends (b) at a €20 limit for three months, with a trigger to revisit after 10,000 live refunds or any confirmed abuse.
1.6 Errors and limits
- Presenting a single score instead of options and consequences.
- Treating "no failures seen" as "no failures possible".
- Omitting "do not deploy" from the options.
- Decisions with no owner or revisit trigger.
Each error weakens a decision. A single score hides the options and the values behind them, such as the finance director's appetite for wrongful refunds. Zero failures in 400 tests still allows a rate near 0.75%. Without "do not deploy", the board never sees the cheapest safe option. A decision with no owner or trigger is never revisited, even after 10,000 live refunds. Check each record for all four.
Exercises
- EX001 — Decision record. Write a seven-part record for whether to let an LLM assistant summarise HR complaints. Rubric: 3 = all seven parts with labelled evidence and triggers; 2 = five or six; 1 = three or four; 0 = fewer.
- EX002 — Facts and values. Classify five statements from §1.5 as fact, estimate, assumption or value. Rubric: 3 = all correct with reasons; 2 = four; 1 = two or three; 0 = fewer.
- EX003 — Bounding. Zero out-of-policy refunds were seen in 1,000 trials. What is the approximate 95% upper bound, and what does it mean for 60,000 refunds a year? Rubric: 3 = ≈0.3%, ≈180 refunds as worst plausible, caveats on independence/representativeness; 2 = numbers without caveats; 1 = one number; 0 = wrong.
- EX004 — RMF gaps. Name one Govern, Map, Measure and Manage question that Amira's record must answer. Rubric: 3 = four apt questions; 2 = three; 1 = one or two; 0 = none.
- EX005 — Reversibility. Propose two reversible design choices for an agent that can change delivery addresses. Rubric: 3 = two concrete, cheap-to-undo controls with triggers; 2 = two without triggers; 1 = one; 0 = none.
Synthesis
Senior AI-security decisions are choices among imperfect options under uncertainty. Record options, evidence and values separately, bound what tests can show, prefer reversible steps and set triggers to revisit.