◇ OpenCyber Academy
Bachelor's · Chapter 21 · English

Chapter 21 — AI and machine-learning security: system models

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 21 — AI and machine-learning security: system models

Lesson map

Lesson 21 published. Map, example, EX061.

flowchart TD
  N0["Learning outcomes"]
  N1["21.1 Components"]
  N2["21.2 Data flows and trust"]
  N3["21.3 Privileged actions"]
  N4["21.4 Using the model"]
  N5["21.4b Training-time components"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX061"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["21.1 Components"]
  N2["21.2 Data flows and trust"]
  N3["21.3 Privileged actions"]
  N4["21.4 Using the model"]
  N5["21.4b Training-time components"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX061"]

[Learning outcomes]
    |
    v
[21.1 Components]
    |
    v
[21.2 Data flows and trust]
    |
    v
[21.3 Privileged actions]
    |
    v
[21.4 Using the model]
    |
    v
[21.4b Training-time components]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX061]

Learning outcomes

You will (1) model an AI system with components, data flows and trust levels; (2) mark untrusted inputs and privileged actions; (3) trace how untrusted content reaches decisions; and (4) use the model to plan controls.

21.1 Components

List users, interfaces, application logic, the model, retrieval stores, tools, external services, training pipelines, model registry and logs. AI systems hide components: a "simple chatbot" may include an embedding model, a vector store and several tools.

Ask Lumen illustrates this. Staff described it as "the chatbot", but Rina's list had eleven components, including the embedding model used for document search, the store of indexed clinic documents, the booking tool, the opening-hours tool and the conversation log. Each component is a place where an attacker might act or data might leak.

21.2 Data flows and trust

For each flow, mark the trust level of its content. Patient messages: authenticated but untrusted content. Clinic documents: internal, medium trust. Partner uploads: external, low trust. Tool results: depend on the tool. Any low-trust content that reaches the model can influence its outputs.

In Rina's diagram, partner uploads are drawn in red, since an outside organisation controls their content. The diagram showed that a red flow reached the booking tool through the model, which is why the confirmation and policy check sit on that path rather than earlier in the conversation.

21.3 Privileged actions

Mark actions with impact: booking, cancelling, sending messages, changing records. Draw the paths from low-trust inputs to these actions; each path needs a control outside the model.

For Ask Lumen, the privileged actions are booking and cancelling appointments. Rina traced three paths to them: from patient messages, from clinic documents and from partner uploads. All three passed through the model, so all three were covered by one external control, the policy check with patient confirmation, rather than three separate filters that might each miss something.

21.4 Using the model

The system model guides threat analysis, control placement and testing. Keep it versioned; update it when tools, data sources or models change.

Rina stores the diagram in the same repository as Ask Lumen's code, so a change that adds a tool must also update the diagram in the same review. When the opening-hours tool was changed, the review included the diagram change, and the reviewer could see the untrusted comments flow had been removed. An outdated diagram gives false comfort.

21.4b Training-time components

System models often stop at the deployed application, but training pipelines matter too. Show where training data comes from, who can add or relabel it, where models are stored and how they reach production. Poisoning and backdoor risks live on these paths, and the same trust-level marking applies: a labelling contractor's edits are lower trust than an internal expert's review.

For DocSort, this view showed the contractor team between the letter archive and the training job. Nothing recorded which contractor labelled which letter, so a poisoned batch could not be traced, a gap the next chapter closes.

21.5 Worked example (synthetic LCH case)

Rina's system model of Ask Lumen showed a forgotten flow: the chatbot could read the clinic's public web page through a "check opening hours" tool, and that page included a comments section. Comments were untrusted public content flowing into the model. LCH changed the tool to read a structured opening-hours file instead of the whole page.

Nobody had listed the tool's data source because the tool seemed harmless; drawing the flow made the risk visible. Reading a structured file also made answers more reliable, since staff edit it directly.

21.6 Errors and limits

A diagram without trust levels cannot show where controls belong, because every flow looks the same. Leaving out tools hides the paths to privileged actions, as with the opening-hours tool that read public comments. A diagram that is not updated with code changes describes a system that no longer exists, so reviews based on it miss new risks introduced since it was drawn. Rina's rule ties diagram updates to code review.

Exercises

Synthesis

A system model with trust levels shows exactly where untrusted content can reach consequential actions — the map for every later control.

Continue studying — chapter 21 of 40

← Chapter 20 (draft) · Course chapter index · Chapter 22 (draft) →