Chapter 20 — AI and machine-learning security: engineering context
Lesson map
Lesson 20 published. Map, example, EX058.
flowchart TD
N0["Learning outcomes"]
N1["20.1 The lifecycle and its attack surface"]
N2["20.2 The NIST AI RMF as an organising frame"]
N3["20.3 Which framework for which question"]
N4["20.4 Scoping"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX058"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["20.1 The lifecycle and its attack surface"]
N2["20.2 The NIST AI RMF as an organising frame"]
N3["20.3 Which framework for which question"]
N4["20.4 Scoping"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX058"]
[Learning outcomes]
|
v
[20.1 The lifecycle and its attack surface]
|
v
[20.2 The NIST AI RMF as an organising frame]
|
v
[20.3 Which framework for which question]
|
v
[20.4 Scoping]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX058]
Learning outcomes
You will (1) describe the AI lifecycle (data, training, evaluation, deployment, monitoring, retirement) and its attack surfaces; (2) use the NIST AI RMF functions to organise engineering work; (3) situate frameworks — NIST AI 100-2 E2025, OWASP LLM 2025, MITRE ATLAS, ISO/IEC 42001, EU AI Act — by purpose; and (4) define the scope of an AI-security engineering task.
20.1 The lifecycle and its attack surface
Data collection can be poisoned; labelling can be corrupted; training pipelines and third-party models can carry backdoors; evaluation can be gamed or leaked; deployed models face evasion, extraction, membership inference and prompt injection; monitoring can be blinded; retired models may leave data behind. Each stage needs owners and controls.
Laying the stages out shows that most attacks need no access to the deployed model at all: a poisoned label or a swapped third-party file does its damage long before deployment. Naming an owner for each stage prevents the common gap where data teams assume security handles it and security teams assume data quality is someone else's job.
20.2 The NIST AI RMF as an organising frame
NIST AI 100-1 defines four functions. Govern: policies, roles, accountability. Map: context, intended use, stakeholders, risks. Measure: analyse, assess and track risks with appropriate methods. Manage: prioritise and act on risks. NIST AI 600-1 profiles these for generative AI and lists risks such as information security, confabulation and data privacy with suggested actions. Engineers mostly work in Map, Measure and Manage, inside Govern's rules.
The functions repeat rather than run once: measurements feed management decisions, which change the system and require new measurement.
20.3 Which framework for which question
- What attacks exist? NIST AI 100-2 E2025 (taxonomy and terminology) and MITRE ATLAS (tactics, techniques, case studies).
- What goes wrong in LLM applications? OWASP Top 10 for LLM Applications 2025.
- How do we manage AI in an organisation? ISO/IEC 42001:2023 (AI management system) and the AI RMF.
- What does the law require? The EU AI Act (Regulation (EU) 2024/1689) for systems in its scope.
Frameworks give vocabulary and checklists; they are not evidence that a system is secure.
Using the wrong framework for a question produces gaps, as when an organisational standard is read as a list of attacks.
20.4 Scoping
An engineering task needs a scope: which system, which lifecycle stages, which threats, what assets and what is out of scope. Explicit exclusions prevent false assurance.
A scope also sets expectations. If model extraction is out of scope, a later report cannot be read as saying extraction was tested and found impossible. Each exclusion carries a reason and a date for review, so it does not quietly become permanent. Sam's scopes also list assumptions, such as 'no external API is exposed', because an exclusion that rests on an assumption must be revisited as soon as the assumption changes.
20.5 Worked example (synthetic NRS case)
NRS asked Sam to "secure the AI". Sam turned it into three scoped workstreams: TrackEye integrity (poisoning via labelling contractor; physical adversarial markings; supply-chain of model updates), Wrench agency (prompt injection; excessive agency; sensitive disclosure), and Pulse resilience (sensor spoofing; feedback-loop drift). Model extraction was explicitly out of scope for this quarter because no AI API is exposed externally.
Each workstream follows the assets and threats of one system, so owners and success measures are clear. The extraction exclusion rests on an assumption, no external API, which Sam lists with a trigger: if NRS ever exposes an AI service externally, extraction returns to scope.
20.6 Errors and limits
- Using a framework checklist as proof of security.
- Scopes with no exclusions.
- Ignoring early lifecycle stages (data, labelling).
A completed checklist shows that topics were considered, not that controls work; evidence comes from tests and measurements. A scope with no exclusions implies that everything was examined, which is rarely true. Early lifecycle stages are easy to ignore because they feel like data management rather than security, yet poisoning and labelling attacks start there and are the hardest to find later, once a model has learned from the bad data.
Exercises
- EX058 — Lifecycle. For each of five lifecycle stages, name one attack. Rubric: 3 = five correct pairs; 2 = four; 1 = two; 0 = none.
- EX059 — RMF. Assign three of Sam's activities to AI RMF functions. Rubric: 3 = three correct with reasons; 2 = two; 1 = one; 0 = none.
- EX060 — Framework choice. Which framework answers "what LLM-application risks should our design review cover?" and why? Rubric: 3 = OWASP LLM 2025 (plus ATLAS/600-1 as complements), reason; 2 = name only; 1 = mismatched; 0 = none.
Synthesis
AI-security engineering starts with the lifecycle, an organising frame and the right framework for each question — then a scope that says what is in and out.
Continue studying — chapter 20 of 50
← Chapter 19 (draft) · Course chapter index · Chapter 21 (draft) →