◇ OpenCyber Academy
Bachelor's · Chapter 19 · English

Chapter 19 — AI and machine-learning security: scope and theory

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 19 — AI and machine-learning security: scope and theory

Lesson map

Lesson 19 published. Map, example, EX055.

flowchart TD
  N0["Learning outcomes"]
  N1["19.1 Definitions"]
  N2["19.2 The taxonomy"]
  N3["19.3 Why ML is vulnerable"]
  N4["19.4 Frameworks"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX055"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["19.1 Definitions"]
  N2["19.2 The taxonomy"]
  N3["19.3 Why ML is vulnerable"]
  N4["19.4 Frameworks"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX055"]

[Learning outcomes]
    |
    v
[19.1 Definitions]
    |
    v
[19.2 The taxonomy]
    |
    v
[19.3 Why ML is vulnerable]
    |
    v
[19.4 Frameworks]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX055]

Learning outcomes

You will (1) define AI security and distinguish it from AI safety and responsible AI; (2) describe the attack classes in the NIST AI 100-2 E2025 taxonomy; (3) explain why machine learning is vulnerable in principle; and (4) place the main frameworks.

19.1 Definitions

AI security protects AI systems and the processes around them from deliberate attack and misuse, and protects others from AI-enabled attack. AI safety concerns harmful behaviour even without an attacker. Responsible AI covers fairness, transparency and accountability. The areas overlap: a prompt injection is a security attack that may cause unsafe or unfair outcomes.

At LCH the three meet in one system. A partner document that makes Ask Lumen announce a false clinic closure is an attack, so it belongs to security; the patients who then miss care are a safety outcome. Rina keeps the terms separate in her reports so each team knows which problems are theirs.

19.2 The taxonomy

NIST AI 100-2 E2025 classifies attacks on predictive AI by objective — availability breakdown, integrity violation, privacy compromise — and adds misuse for generative AI. Attack classes include evasion (test-time inputs), poisoning (training data or model), privacy attacks (membership inference, data reconstruction, model extraction) and, for generative systems, direct and indirect prompt injection. Attackers are characterised by knowledge (white-, grey-, black-box) and capabilities (training-data control, model control, query access).

Rina places each of LCH's systems in it. Ask Lumen is generative, so prompt injection and misuse apply; NoShow and DocSort are predictive. An outside attacker usually has only black-box query access, while a contractor who labels DocSort letters has training-data control, which is a stronger capability.

19.3 Why ML is vulnerable

Models learn statistical patterns from data, not rules; they generalise imperfectly and are sensitive to small, carefully chosen input changes (Szegedy et al., arXiv:1312.6199; Goodfellow et al., arXiv:1412.6572). They inherit whatever their data contains, including poison. Large language models follow instructions found anywhere in their input. These properties are inherent, so security relies on layered controls rather than a single fix.

The evasion lab shows this concretely: the same toy classifier that scores 0.985 on clean inputs falls to about chance once the perturbation budget reaches one, with no change to the model itself.

19.4 Frameworks

NIST AI RMF (AI 100-1) and its Generative AI Profile (AI 600-1) organise risk management; OWASP LLM 2025 lists application risks; MITRE ATLAS catalogues adversary techniques with case studies; ISO/IEC 42001 specifies an AI management system; the EU AI Act sets legal duties for some systems.

They answer different questions. Rina uses OWASP to check Ask Lumen's application risks, ATLAS to find real attack examples for her staff briefing, and the AI RMF to structure LCH's overall risk process. Whether the EU AI Act applies depends on each system's use, so she records that assessment separately rather than assuming one answer covers all three systems.

19.5 Worked example (synthetic LCH case)

Rina classified LCH's risks: Ask Lumen — indirect prompt injection via documents (integrity), sensitive disclosure (privacy); NoShow — membership inference (privacy), manipulation of patient-controlled fields (integrity, evasion); DocSort — crafted letters to misroute (evasion), poisoning via relabelled training letters (integrity).

The classification showed that each system had both privacy and integrity risks, but through different paths. Ask Lumen's main exposure was what it reads; NoShow's was what it reveals; DocSort's was who shaped its training data. That difference led Rina to assign different owners: the chatbot team, the data protection officer and the records team.

19.6 Errors and limits

Focusing on jailbreaks would have missed DocSort's poisoning risk entirely, since nobody chats with DocSort. No single control covers evasion, poisoning and injection, so LCH needs several layers. Using "safety" where "security" is meant leaves it unclear whether an attacker is assumed, which changes the tests that should be run and who is responsible for fixing the problem. Rina's reports keep the terms apart, so a false clinic closure caused by a partner document is filed as an attack and the patients' missed care as a safety outcome.

Exercises

Synthesis

AI security has a precise scope and shared vocabulary. Its vulnerabilities are structural, which is why the rest of the book builds layered, measured defences.

Continue studying — chapter 19 of 40

← Chapter 18 (draft) · Course chapter index · Chapter 20 (draft) →