◇ OpenCyber Academy
Engineering · Chapter 18 · English

Chapter 18 — Security architecture

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 18 — Security architecture

Lesson map

Lesson 18 published. Map, example, EX052.

flowchart TD
  N0["Learning outcomes"]
  N1["18.1 Principles"]
  N2["18.2 Trust boundaries"]
  N3["18.3 A reference architecture for agents"]
  N4["18.4 Architecture decision records"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX052"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["18.1 Principles"]
  N2["18.2 Trust boundaries"]
  N3["18.3 A reference architecture for agents"]
  N4["18.4 Architecture decision records"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX052"]

[Learning outcomes]
    |
    v
[18.1 Principles]
    |
    v
[18.2 Trust boundaries]
    |
    v
[18.3 A reference architecture for agents]
    |
    v
[18.4 Architecture decision records]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX052]

Learning outcomes

You will (1) apply architecture principles (least privilege, separation of duties, complete mediation, fail-safe defaults, defence in depth) to AI systems; (2) define trust boundaries for models, data and tools; (3) design a reference architecture for an LLM agent; and (4) document architecture decisions with threats and residual risk.

18.1 Principles

Saltzer and Schroeder's classic principles still apply. Least privilege: the model and its tools get only the permissions needed. Complete mediation: every action is checked by a policy point, every time. Fail-safe defaults: deny unless explicitly allowed. Separation of duties: the component that proposes an action is not the one that approves it. Economy of mechanism: keep the policy small enough to review.

These principles predate machine learning, which is useful: they give reviewers a checklist that does not depend on how the model behaves. A model can be manipulated, but a policy point that checks every action still applies whatever the model was persuaded to propose.

18.2 Trust boundaries

A trust boundary is where data or control passes between parties with different trust. In LLM applications the critical boundary runs inside the prompt: system instructions (trusted), user input (partly trusted), retrieved or tool-returned content (untrusted). Because the model cannot reliably enforce this boundary, the architecture must enforce consequences outside the model.

The model sees all three kinds of text as one sequence of tokens. Delimiters and instructions help it separate them, but a carefully written bulletin can still be read as an instruction. Enforcing consequences outside means the policy service, tool credentials and human approval decide what actually happens, so the model's confusion about whose text it is reading cannot by itself cause harm.

18.3 A reference architecture for agents

Components: (1) an ingestion service that labels provenance; (2) the model, which proposes actions; (3) a policy decision point that checks each proposal against identity, scope and rules; (4) tool adapters with narrow credentials; (5) human approval for high-impact actions; (6) logging and monitoring. Research on structured queries (Chen et al., arXiv:2402.06363) improves the model's own separation of instructions and data but does not replace the external policy point. OWASP LLM06 (excessive agency) warns against excessive functionality, permissions and autonomy.

Each component maps to a principle: the policy point gives complete mediation, narrow credentials give least privilege, and human approval separates proposing from approving.

18.4 Architecture decision records

Record each significant decision: context, options considered, decision, consequences, threats addressed and residual risk. Reviewers and future engineers can then see why a control exists.

Without a record, a control that looks inconvenient may be removed by someone who does not know the attack it prevents. Records are short, usually a page, and stored with the code so they change through the same review. When conditions change, such as a new tool or a new threat, the record is revisited and a new one supersedes it, keeping the history. Residual risk is the field most often left blank and the one reviewers most need.

18.5 Worked example (synthetic NRS case)

Sam wrote ADR-007: "Wrench work-order creation". Options: (a) model calls API directly; (b) model proposes, policy checks, technician confirms; (c) no tool, model drafts text for the technician to copy. Decision: (b). Consequences: one more click for technicians; injected instructions cannot create orders without a human. Residual risk: technicians may approve without reading — mitigated by showing the source document of any instruction-like text.

Option (c) was safest but gave up the benefit of structured orders, while option (a) put an untrusted model in direct control of a privileged system. The record shows why (b) sits between them.

18.6 Errors and limits

A boundary enforced only by instructions in the prompt holds until a clever input breaks it. Broad credentials turn a single manipulated call into access to everything the credential reaches. Decisions recorded without residual risk give the impression that a problem is solved when it is only reduced. Sam's architecture review asks of each design: where is the boundary enforced in code, what can each tool's credential do, and what risk remains.

Exercises

Synthesis

Architecture is where AI security becomes enforceable: trust boundaries, an external policy point, least-privilege tools and documented decisions turn model uncertainty into bounded risk.

Continue studying — chapter 18 of 50

← Chapter 17 (draft) · Course chapter index · Chapter 19 (draft) →