Chapter 10 — AI and machine-learning security: orientation
Lesson map
Lesson 10 published. Map, example, EX028.
flowchart TD
N0["Learning outcomes"]
N1["10.1 How ML systems are built"]
N2["10.2 Attack classes"]
N3["10.3 OWASP and ATLAS"]
N4["10.4 The technologist's role"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX028"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["10.1 How ML systems are built"]
N2["10.2 Attack classes"]
N3["10.3 OWASP and ATLAS"]
N4["10.4 The technologist's role"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX028"]
[Learning outcomes]
|
v
[10.1 How ML systems are built]
|
v
[10.2 Attack classes]
|
v
[10.3 OWASP and ATLAS]
|
v
[10.4 The technologist's role]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX028]
Learning outcomes
You will (1) explain how machine-learning systems are built and used; (2) list the main attack classes on AI; (3) relate them to OWASP LLM Top 10 2025 and MITRE ATLAS; and (4) describe the technologist's role in AI security.
10.1 How ML systems are built
A machine-learning system learns a function from data. The lifecycle runs: collect and label data, train a model, evaluate it, deploy it behind an application, monitor it, and retrain. Language-model applications usually start from a pre-trained model from a vendor, then add a system prompt, retrieved documents and tools. Brightwell's three systems show both patterns. ReturnGuard follows the classic lifecycle, trained on labelled returns and retrained as fraud changes, while Ask Brightwell and CodePilot start from vendor or pre-trained models and add prompts, documents and tools. The security work differs accordingly: data and labels for the first, prompts and tools for the others.
10.2 Attack classes
NIST AI 100-2 E2025 groups attacks into: evasion — inputs crafted to cause wrong outputs (Szegedy et al., arXiv:1312.6199; Goodfellow et al., arXiv:1412.6572); poisoning — corrupting training data, including backdoors (Gu et al., arXiv:1708.06733); privacy — membership inference, data extraction and model extraction (Shokri et al., arXiv:1610.05820; Tramèr et al., arXiv:1609.02943); and, for generative AI, direct and indirect prompt injection and misuse. Jonas's map of Brightwell uses these classes: ReturnGuard faces evasion by fraudsters shaping returns, poisoning through wrong labels, and extraction through repeated scoring, while Ask Brightwell faces indirect prompt injection through reviews and pages. Naming the class helps choose the lab and the control.
10.3 OWASP and ATLAS
The OWASP Top 10 for LLM Applications 2025: LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure, LLM03 Supply Chain, LLM04 Data and Model Poisoning, LLM05 Improper Output Handling, LLM06 Excessive Agency, LLM07 System Prompt Leakage, LLM08 Vector and Embedding Weaknesses, LLM09 Misinformation, LLM10 Unbounded Consumption. MITRE ATLAS describes adversary tactics and techniques against AI, with case studies. For Brightwell, the OWASP list fits Ask Brightwell and CodePilot, while ATLAS also covers ReturnGuard, which is a predictive model that the OWASP LLM list does not address. Jonas uses both so that every system has a recognised name for each risk.
10.4 The technologist's role
Technologists build and run the systems around models: gateways, orchestrators, retrieval, tools, pipelines, monitoring. Most effective AI-security controls live there: access control, validation, isolation, limits, logging. You do not need to train models to secure them. At Brightwell, Jonas's team does not train any language model, yet nearly every fix in this book is theirs: the rate limit, the index split, the ownership checks, the policy service, the sandbox and the monitoring. The models stay the same; what changes is how much they are allowed to reach. This is why the book spends most chapters on systems around models rather than on training.
10.5 Worked example (synthetic Brightwell case)
Jonas maps Brightwell's three systems. Ask Brightwell: LLM01, LLM02, LLM05, LLM06, LLM08, LLM10. ReturnGuard: evasion (fraudsters shaping returns), poisoning (wrong labels from staff), model extraction via repeated scoring. CodePilot: LLM03 (suggested packages), LLM05 (generated code run without review). He notes that most fixes are in his team's code, not the models. The map gives each system a short list of tests to run. Its last observation matters most for Jonas's team: since the fixes are in their code, they can make them without waiting for a vendor.
10.6 Errors and limits
- Assuming AI security means model training.
- Treating OWASP LLM as covering predictive models.
- Forgetting the data pipeline.
Jonas's map avoids these by listing what each system actually does. ReturnGuard is a predictive model and needs evasion, poisoning and extraction thinking, not only the LLM list. The labels for ReturnGuard come from staff, so the data pipeline is part of its attack surface. And the controls he proposes are all in code his team owns, not in model training. Each error leaves part of a system's attack surface unexamined.
Exercises
- EX028 — Lifecycle. Name one attack at each lifecycle stage. Rubric: 3 = five stages with apt attacks; 2 = four; 1 = two; 0 = none.
- EX029 — Classify. Classify four Brightwell scenarios as evasion, poisoning, privacy or injection. Rubric: 3 = four correct; 2 = three; 1 = one or two; 0 = none.
- EX030 — Role. List five AI-security controls a technologist owns. Rubric: 3 = five apt; 2 = four; 1 = two; 0 = none.
Synthesis
AI security covers the whole lifecycle and many attack classes; technologists own most of the controls that make AI systems safe to run.
Continue studying — chapter 10 of 30
← Chapter 09 (draft) · Course chapter index · Chapter 11 (draft) →