◇ OpenCyber Academy
Engineering · Chapter 15 · English

Chapter 15 — Security fundamentals

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 15 — Security fundamentals

Lesson map

Lesson 15 published. Map, example, EX043.

flowchart TD
  N0["Learning outcomes"]
  N1["15.1 CIA for AI"]
  N2["15.2 Threat, vulnerability, risk, control"]
  N3["15.3 Defence in depth"]
  N4["15.4 The attacker's knowledge and capabilities"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX043"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["15.1 CIA for AI"]
  N2["15.2 Threat, vulnerability, risk, control"]
  N3["15.3 Defence in depth"]
  N4["15.4 The attacker's knowledge and capabilities"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX043"]

[Learning outcomes]
    |
    v
[15.1 CIA for AI]
    |
    v
[15.2 Threat, vulnerability, risk, control]
    |
    v
[15.3 Defence in depth]
    |
    v
[15.4 The attacker's knowledge and capabilities]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX043]

Learning outcomes

You will (1) apply confidentiality, integrity and availability (CIA) to AI assets; (2) describe threat, vulnerability, risk and control precisely; (3) use defence in depth and least privilege; and (4) map AI-specific attacks into the NIST AI 100-2 E2025 taxonomy.

15.1 CIA for AI

Confidentiality: training data, prompts, model weights and outputs. Integrity: data, weights, retrieval content, model decisions and tool actions. Availability: inference capacity and dependent business processes. NIST AI 100-2 E2025 organises adversarial attacks on predictive AI by the attacker's objective — availability breakdown, integrity violation, privacy compromise — and adds misuse enablement for generative AI.

Listing assets this way exposes items that ordinary security reviews miss. Prompts may contain business logic or secrets. Weights may be worth stealing. Tool actions, such as a work order created in Wrench's name, carry integrity requirements because they change the physical world. Availability extends beyond the model to the work that depends on it, such as track inspection.

15.2 Threat, vulnerability, risk, control

A threat is a potential cause of harm (an actor or event). A vulnerability is a weakness it can exploit. Risk combines likelihood and impact. A control reduces likelihood or impact. For Wrench: threat — a malicious supplier; vulnerability — the model follows instructions in bulletins; impact — false work orders; control — human confirmation of work orders.

Writing risk as a chain shows where a control acts. Human confirmation does not stop the model following instructions; it limits the impact, because a false work order needs a person's approval before it takes effect. A second control, such as provenance labels, could reduce the likelihood.

15.3 Defence in depth

No single control is perfect, especially for models whose behaviour is statistical. Layer controls: input provenance labelling, model-level instructions, output validation, policy enforcement outside the model, least-privilege tools, monitoring and human oversight. Each layer catches some failures of the others.

The layers must be independent to add value. If output validation and policy enforcement both rely on the model's own judgement, one successful manipulation defeats both. Wrench's design keeps policy in code outside the model and requires a human confirmation that the model cannot supply, so each layer fails for different reasons. Sam checks this by asking, for each layer, what an attacker would need in order to bypass it.

15.4 The attacker's knowledge and capabilities

The taxonomy also classifies attackers by knowledge (white-box, grey-box, black-box) and capabilities (control of training data, of the model, of queries, of the source code). Stating these turns vague fears into testable threat models.

For TrackEye, a realistic attacker is black-box with physical access: anyone walking the line can place markings, but cannot read the model. A labelling contractor has training-data capability. Each profile becomes a test campaign with its own methods, and results are reported per profile, since a model robust to one may be weak against another.

15.5 Worked example (synthetic NRS case)

Sam built a CIA table for TrackEye. Confidentiality: low (images of rails are not secret) but model weights are commercially valuable. Integrity: critical — a missed defect is a safety risk; poisoning through the labelling contractor and physical markings were the main threats. Availability: moderate — if TrackEye is down, manual inspection continues at higher cost. The table steered effort toward integrity controls.

Integrity is rated critical because its impact is safety, not money. Availability is only moderate because a manual fallback exists, though attacks forcing frequent fallback would raise costs.

15.6 Errors and limits

Prompt injection matters, but treating it as the whole field ignores poisoning, model theft, privacy leaks and resource abuse. Designs with a single control fail completely when that control is bypassed. Risk statements that name a threat without its impact cannot be prioritised, because a weakness that leaks rail images and one that hides a cracked rail look equally urgent. Sam's risk register therefore requires both likelihood and impact for every entry.

Exercises

Synthesis

Security fundamentals apply directly to AI: CIA for new assets, precise risk language, layered controls and explicit attacker capabilities.

Continue studying — chapter 15 of 50

← Chapter 14 (draft) · Course chapter index · Chapter 16 (draft) →