Chapter 21 — AI and machine-learning security: requirements
Lesson map
Lesson 21 published. Map, example, EX061.
flowchart TD
N0["Learning outcomes"]
N1["21.1 From abuse cases to requirements"]
N2["21.2 Kinds of requirement"]
N3["21.3 Writing verifiable statistical requirements"]
N4["21.4 Traceability"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX061"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["21.1 From abuse cases to requirements"]
N2["21.2 Kinds of requirement"]
N3["21.3 Writing verifiable statistical requirements"]
N4["21.4 Traceability"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX061"]
[Learning outcomes]
|
v
[21.1 From abuse cases to requirements]
|
v
[21.2 Kinds of requirement]
|
v
[21.3 Writing verifiable statistical requirements]
|
v
[21.4 Traceability]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX061]
Learning outcomes
You will (1) write security requirements for AI components that are specific, measurable and verifiable; (2) derive requirements from abuse cases; (3) express statistical requirements with confidence bounds; and (4) trace requirements to frameworks and regulation.
21.1 From abuse cases to requirements
An abuse case describes how someone could misuse the system: "A supplier embeds an instruction in a bulletin that causes Wrench to schedule unnecessary maintenance." Each abuse case yields requirements that prevent, detect or limit it.
Writing abuse cases as stories with an actor, a goal and a route makes them concrete enough to test. The bulletin case gives three requirements: label bulletin text as untrusted (prevent), alert on instruction-like content in bulletins (detect), and require confirmation for work orders (limit). Having all three means one failing does not leave the abuse case open.
21.2 Kinds of requirement
- Functional security: "Work orders require technician confirmation."
- Statistical behaviour: "On the held-out injection suite (≥ 600 cases), Wrench proposes unauthorised tool calls in ≤ 2% of cases (95% upper bound)."
- Process: "Every model release passes the security evaluation gate."
- Data: "Training images carry provenance; contractor relabels are sampled and reviewed at 5%."
- Operational: "Per-user action budget of 20 work orders per hour; alerts above 3× baseline page the on-call."
The 600-case minimum is twice the 300-case suite that blocked the friendlier prompt, because a smaller suite gives an upper bound too wide to show a rate near 2%.
21.3 Writing verifiable statistical requirements
State the population (test suite), metric, threshold, confidence level and minimum sample size. "Robust to attacks" is not verifiable. "Accuracy under PGD with L∞ ε = 4/255 ≥ 70% on the held-out set" is — and it states the threat model implicitly.
The PGD example names the attack, the norm and the budget, so testers know exactly what to run and readers know what the result does not cover: larger budgets, other norms and physical attacks. A statistical requirement also needs enough data to decide. If the test set is too small, the upper bound may exceed the threshold even when the model is good, and the requirement cannot be shown to pass. The sample size is therefore part of the requirement itself.
21.4 Traceability
Link each requirement to its source (abuse case, OWASP entry, AI Act article, ISO/IEC 42001 control objective) and its verification method. EU AI Act Article 15 requires high-risk systems to be resilient against attempts to alter use or performance by exploiting vulnerabilities, naming data poisoning, model poisoning, adversarial examples and confidentiality attacks; requirements addressing each make compliance arguments traceable.
A trace table runs both ways: from each source to the requirements that answer it, and from each requirement to its test. Gaps then appear as empty cells.
21.5 Worked example (synthetic NRS case)
Sam wrote REQ-AI-014: "TrackEye false-negative rate for defects ≥ 10 mm shall be ≤ 1.5% with 95% upper bound, per condition stratum (day, dawn/dusk, night, rain), on a held-out set of ≥ 1,000 defects per stratum; verified by test report at each release." Source: hazard H-03 and AI Act Art. 15 (accuracy). REQ-AI-015 addressed adversarial markings: "No single marking ≤ 20 × 20 cm placed ≥ 0.5 m from the rail shall reduce detection of an adjacent defect below 95% in the physical test campaign."
The stratum rule closes the dawn gap: a good overall rate can no longer hide a weak condition.
21.6 Errors and limits
- Requirements without thresholds or test sets.
- Thresholds without confidence bounds.
- Requirements that cannot be verified with available data.
Without a threshold and a test set, nobody can say whether a requirement has been met. A threshold without a confidence bound can be passed by luck on a small sample. A requirement needing data that does not exist, such as a thousand night-time defects per stratum when only two hundred have been recorded, must either be rewritten or come with a plan to collect the data before the release that depends on it.
Exercises
- EX061 — Rewrite. Rewrite "Wrench must not leak secrets" as a verifiable requirement. Rubric: 3 = population, metric, threshold, confidence, verification; 2 = four; 1 = vague; 0 = unchanged.
- EX062 — Abuse case. Write an abuse case for Pulse and one derived requirement. Rubric: 3 = plausible case and verifiable requirement; 2 = one weak; 1 = vague; 0 = none.
- EX063 — Sample size. How many test cases are needed to show a rate ≤ 2% with a 95% one-sided upper bound if zero failures are observed? Rubric: 3 = about 150 (rule of three: 3/n ≤ 0.02); 2 = method only; 1 = wrong but reasoned; 0 = none.
Synthesis
Good AI-security requirements are measurable statements about behaviour under stated attacks, with sample sizes and confidence, traced to their source and to their verification.
Continue studying — chapter 21 of 50
← Chapter 20 (draft) · Course chapter index · Chapter 22 (draft) →