◇ OpenCyber Academy
Technologist · Chapter 18 · English

Chapter 18 — AI and machine-learning security: controls

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 18 — AI and machine-learning security: controls

Lesson map

Lesson 18 published. Map, example, EX052.

flowchart TD
  N0["Learning outcomes"]
  N1["18.1 Policy service"]
  N2["18.2 Input controls"]
  N3["18.3 Output controls"]
  N4["18.4 Human confirmation"]
  N5["18.5 Verifying controls"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX052"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["18.1 Policy service"]
  N2["18.2 Input controls"]
  N3["18.3 Output controls"]
  N4["18.4 Human confirmation"]
  N5["18.5 Verifying controls"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX052"]

[Learning outcomes]
    |
    v
[18.1 Policy service]
    |
    v
[18.2 Input controls]
    |
    v
[18.3 Output controls]
    |
    v
[18.4 Human confirmation]
    |
    v
[18.5 Verifying controls]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX052]

Learning outcomes

You will (1) implement a policy service for agent tools; (2) configure input and output controls; (3) implement human confirmation correctly; and (4) verify each control with tests.

18.1 Policy service

The policy service receives a proposed action (tool, parameters, user identity, conversation type) and returns allow or deny with a reason. Rules are code: "start_return allowed only if order.owner == user and order.status == delivered and today − delivered_date ≤ 30 days and value ≤ €1,000". The model never sees or bypasses this logic. Keep rules small and readable so reviewers can check them, and log every decision with its reason. The rule mirrors Brightwell's published 30-day returns policy, so the service enforces the same promise customers read on the site.

18.2 Input controls

Limit message length; restrict retrieval sources; mark untrusted content with delimiters; optionally run an injection classifier as a signal. Structured-query approaches such as StruQ (arXiv:2402.06363) separate instructions from data at the model interface; they reduce but do not remove risk. Input controls lower how often bad outputs occur; they do not bound the damage when one gets through. For Ask Brightwell, input controls are the gateway's length limit, retrieval restricted to public documents without reviews, and delimiters around retrieved text. The separated prompt reduced harmful proposals in Jonas's study, but the policy service remains the control that bounds damage when an injection gets through.

18.3 Output controls

Validate structured actions; escape text for its destination; strip or allow-list links and images in answers (to prevent data leaking through URLs); check answers about prices or policies against source documents where feasible. For Ask Brightwell, answers may contain links only to Brightwell's own domain, and images are not rendered, because an injected instruction could otherwise hide customer data in a URL. Answers about the return window are compared with the current policy page, which would have caught the expired 60-day campaign page sooner. Output checks are tested with planted cases like any other control.

18.4 Human confirmation

Show the user exactly what will happen, built from structured data: "Start a return for order 100245 — Oslo sofa, €899 — refund to card ending 4421? Yes / No". Do not show model-written summaries in confirmation, since injected text can make harmful actions look harmless. Keep confirmations rare enough that people still read them. Ask Brightwell's confirmation uses exactly this format, with values from the order record. Because confirmation appears only for returns and address changes, customers see it rarely and are more likely to read it.

18.5 Verifying controls

Each control gets unit tests (rules allow and deny the right cases) and suite tests (injection cases no longer cause harmful actions). Re-run on every change; a control that silently breaks during a refactor is worse than an absent one, because people rely on it. Ask Brightwell's policy service has 40 unit tests covering edge cases such as delivery exactly 30 days ago, and the held-out injection suite runs nightly against staging. When a developer's prompt change later raised harmful proposals, these tests and the gate caught it before release. Each test names the rule it checks, so a failure tells the developer which rule broke.

18.6 Worked example (synthetic Brightwell case)

After adding the policy service and structured confirmation, Jonas reruns the held-out suite: harmful returns fall from 31 of 80 indirect cases to 0 of 80 (upper bound ≈ 4%); normal returns still complete in 97% of 500 test conversations. He adds 40 unit tests for the policy rules, including edge cases such as orders delivered exactly 30 days ago. The upper bound matters: zero in 80 does not prove zero risk, only that a high rate is unlikely. Measuring normal returns alongside shows the control did not buy safety by refusing everything.

18.7 Errors and limits

Ask Brightwell's design avoids each. The returns rules live in the policy service, not the prompt; the confirmation screen is built from the order record; and every rule has unit tests. A rule written in the prompt can be talked around by a planted review. A control without tests can disappear in a refactor without anyone noticing until the next incident. The confirmation screen now has its own test.

Exercises

Synthesis

Controls that matter are code outside the model — policy services, validators, structured confirmation — each proven by tests.

Continue studying — chapter 18 of 30

← Chapter 17 (draft) · Course chapter index · Chapter 19 (draft) →