Chapter 26 — AI and machine-learning security: architecture alternatives | Engineering AI and machine-learning security

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 26 published. Map, example, EX076.

flowchart TD
  N0["Learning outcomes"]
  N1["26.1 Patterns for LLM applications"]
  N2["26.2 Patterns for predictive ML services"]
  N3["26.3 Hosted versus self-hosted"]
  N4["26.4 Trade-off analysis"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX076"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["26.1 Patterns for LLM applications"]
  N2["26.2 Patterns for predictive ML services"]
  N3["26.3 Hosted versus self-hosted"]
  N4["26.4 Trade-off analysis"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX076"]

[Learning outcomes]
    |
    v
[26.1 Patterns for LLM applications]
    |
    v
[26.2 Patterns for predictive ML services]
    |
    v
[26.3 Hosted versus self-hosted]
    |
    v
[26.4 Trade-off analysis]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX076]

Learning outcomes

You will (1) compare architectural patterns for LLM applications and ML services by security properties; (2) evaluate patterns such as dual-model (privileged/quarantined), plan-then-execute, retrieval isolation and human-in-the-loop; (3) choose between hosted and self-hosted models with a threat lens; and (4) justify a choice with a trade-off table.

26.1 Patterns for LLM applications

Single model with tools: simplest, highest injection exposure because untrusted text and tool authority meet in one context.
Dual model: a quarantined model reads untrusted content and returns only structured, constrained outputs (for example a category label or extracted numbers); a privileged model plans actions using those outputs but never sees raw untrusted text.
Plan-then-execute: the model commits to a plan from the user's request before reading untrusted data; later content can fill in parameters but not add actions.
Retrieval isolation: sensitive indexes are separated and filtered by user identity before the model sees anything.
Human-in-the-loop: high-impact actions require approval, with provenance shown.

These patterns reduce injection impact by limiting what untrusted text can influence; none removes it entirely.

26.2 Patterns for predictive ML services

Return labels rather than full probability vectors when clients do not need confidences (reduces extraction and membership-inference signal); add input validation and out-of-distribution detection; ensemble or randomise where it increases attack cost; isolate training from serving.

Each measure raises the cost of a specific attack. Returning only labels removes the fine-grained scores that extraction and membership inference rely on, as the membership lab showed with confidence outputs. Out-of-distribution detection flags inputs unlike the training data, which includes many adversarial ones. Isolating training from serving means a compromised serving host cannot alter the data the next model learns from.

26.3 Hosted versus self-hosted

Hosted models offer strong operations and fast updates but change without notice, process data outside your boundary and may restrict testing. Self-hosted models give control and reproducibility but make you responsible for patching, capacity and safety tuning. The choice is a risk decision involving data protection, testing authority and supply-chain trust.

For NRS, sending rail-maintenance documents to an outside provider may be acceptable, while sending HR reports may not; the data's classification decides as much as the model's quality. A hosted model that updates silently can also invalidate a security evaluation overnight, so contracts should state how changes are announced and whether customers may test.

26.4 Trade-off analysis

Score each alternative on residual injection impact, data exposure, utility, latency, cost and operational burden. Make the weighting explicit and get stakeholders to agree on it before scoring.

Agreeing weights first prevents the analysis from being tuned to justify a favourite. Scores need short reasons, such as 'dual model: untrusted text yields only a fixed record, so injected actions cannot reach the planner model'. Where a criterion such as latency can be measured, a measurement replaces the estimate. The result is a recommendation with its basis visible, which stakeholders can challenge on the facts rather than on preference, and which can be rescored if weights change later.

26.5 Worked example (synthetic NRS case)

For the depot-planner assistant, Sam compared single-model, dual-model and plan-then-execute. Dual-model scored best on injection impact (untrusted bulletins only produce a constrained "affected asset IDs and severity" record) but cost 30% more latency. Plan-then-execute could not handle planners' iterative questions well. The team chose dual-model for bulletin processing and single-model for free conversation without tools.

The split decision combines the strengths of both patterns. Free conversation has no tools, so injection there can at worst produce a misleading answer, and bulletins reach planning only as constrained records. The 30% latency cost falls only on bulletin processing, which runs in the background rather than while a planner waits for an answer.

26.6 Errors and limits

Choosing patterns by fashion rather than threat model.
Quarantined models whose "structured" output is free text.
Ignoring hosted-model terms that restrict security testing.

A pattern chosen because others use it may defend against threats the system does not face while missing those it does. A quarantined model that returns free text gives injected instructions a direct route to the privileged model, defeating the pattern's purpose. Hosted-model terms that forbid testing leave a team unable to verify its own security claims, so those terms are checked before the choice is made.

Exercises

EX076 — Dual model. Explain why the quarantined model's output must be constrained. Rubric: 3 = free text could carry injected instructions to the privileged model; constrained schema limits influence; 2 = partial; 1 = vague; 0 = wrong.
EX077 — Labels only. What two attacks become harder when an API returns labels only? Rubric: 3 = extraction and membership inference (confidence-based); 2 = one; 1 = vague; 0 = wrong.
EX078 — Hosted vs self-hosted. Give two security reasons for each choice. Rubric: 3 = two each, accurate; 2 = three total; 1 = one or two; 0 = none.

Synthesis

Architecture choices determine how far untrusted input can reach. Dual-model, plan-then-execute, isolation and approval patterns trade latency and flexibility for bounded impact.

Continue studying — chapter 26 of 50

← Chapter 25 (draft) · Course chapter index · Chapter 27 (draft) →