Chapter 16 — Security architecture
Lesson map
Lesson 16 published. Map, example, EX046.
flowchart TD
N0["Learning outcomes"]
N1["16.1 Principles applied to AI"]
N2["16.2 Trust boundaries in LLM applications"]
N3["16.3 The policy enforcement point"]
N4["16.4 Comparing options"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX046"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["16.1 Principles applied to AI"]
N2["16.2 Trust boundaries in LLM applications"]
N3["16.3 The policy enforcement point"]
N4["16.4 Comparing options"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX046"]
[Learning outcomes]
|
v
[16.1 Principles applied to AI]
|
v
[16.2 Trust boundaries in LLM applications]
|
v
[16.3 The policy enforcement point]
|
v
[16.4 Comparing options]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX046]
Learning outcomes
You will (1) apply least privilege, defence in depth, complete mediation and fail-safe defaults to AI systems; (2) draw trust boundaries in LLM applications; (3) place a policy enforcement point outside the model; and (4) compare architecture options by residual risk.
16.1 Principles applied to AI
Least privilege: give the model and its tools only the permissions needed. Defence in depth: combine input labelling, output validation, policy checks, monitoring and human review. Complete mediation: check every action, every time. Fail-safe defaults: deny when unsure.
For Ask Lumen, least privilege is the delegated token; defence in depth combines labelling of retrieved text, output parsing, the policy check and logging; complete mediation means every booking goes through the policy check, including those proposed late in a long conversation; fail-safe defaults mean an unparseable or doubtful proposal is refused, and the patient is told how to book by phone.
16.2 Trust boundaries in LLM applications
A language model receives trusted instructions (from developers), partly trusted input (from users) and untrusted content (retrieved documents, web pages, tool results) in a single sequence of tokens. It cannot reliably tell them apart, which is why indirect prompt injection works (Greshake et al., arXiv:2302.12173). The architecture must therefore enforce consequences outside the model.
For Ask Lumen, the system prompt is trusted, the patient's message is partly trusted, and retrieved clinic documents, especially partner uploads, are untrusted. Labelling them helps the model but cannot guarantee it ignores instructions inside a document, so the booking tool's own checks are what bound the outcome.
16.3 The policy enforcement point
The model proposes actions; a separate, small, tested component decides whether to execute them based on the user's identity, the action, its parameters and whether untrusted content influenced it. High-impact actions require human confirmation. OWASP LLM06 (excessive agency) is addressed here.
Keeping this component small is deliberate: a few hundred lines of code can be reviewed and tested thoroughly, unlike the model. In Ask Lumen, the policy point compares the booking's patient with the token's subject, checks the clinic and date rules, and requires confirmation for cancellations. Every decision, allowed or refused, is logged with the request ID.
16.4 Comparing options
Options for a booking chatbot range from no tools (answers only) to free tool use. Between them: tools with policy checks and confirmations. Compare by residual risk, usefulness and cost, and record the decision with its reasoning.
Residual risk is what remains after controls; usefulness is how many patients can complete their task; cost includes building and operating each option. A table with these three columns makes the trade-off visible. Recording the reasoning lets a later team understand why the choice was made, and revisit it if conditions change, such as a new tool being added.
16.5 Worked example (synthetic LCH case)
Rina's architecture note for Ask Lumen v2 listed three options: (a) model books directly; (b) model proposes, policy checks patient identity and clinic, patient confirms in the UI; (c) chatbot gives instructions and patients book themselves. LCH chose (b): confirmation adds one click, but injected text can no longer book or cancel without the patient's action.
Option (a) was rejected because a single injected document could act; option (c) because it removed most of the chatbot's value. The research study in the research-methods chapter later measured the cost of (b)'s extra click.
16.6 Errors and limits
- Security rules written only in the system prompt.
- Tools with broader permissions than users.
- No record of why an architecture was chosen.
Rules written only in the system prompt can be overridden by persuasive text, since the model treats both as language. Tools with broader permissions than users let a manipulated model exceed what any user could do. Without a record of why an architecture was chosen, later changes may remove a control without realising what it protected against, such as the confirmation step that stops injected bookings.
Exercises
- EX046 — Principles. Apply three principles to DocSort. Rubric: 3 = three concrete applications; 2 = two; 1 = names only; 0 = none.
- EX047 — Boundary. Why can't the model enforce the trust boundary itself? Rubric: 3 = single token stream, statistical instruction-following; 2 = partial; 1 = vague; 0 = wrong.
- EX048 — Option choice. Argue for option (b) over (a) in two sentences. Rubric: 3 = bounded impact of injection with modest friction; 2 = partial; 1 = vague; 0 = none.
Synthesis
Security architecture moves enforcement out of the model: trust boundaries, an external policy point and human confirmation bound what a manipulated model can do.
Continue studying — chapter 16 of 40
← Chapter 15 (draft) · Course chapter index · Chapter 17 (draft) →