Chapter 22 — AI and machine-learning security: automation
Lesson map
Lesson 22 published. Map, example, EX064.
flowchart TD
N0["Learning outcomes"]
N1["22.1 Pipeline checks"]
N2["22.2 Release gates"]
N3["22.3 Automated containment"]
N4["22.4 Automation risks"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX064"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["22.1 Pipeline checks"]
N2["22.2 Release gates"]
N3["22.3 Automated containment"]
N4["22.4 Automation risks"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX064"]
[Learning outcomes]
|
v
[22.1 Pipeline checks]
|
v
[22.2 Release gates]
|
v
[22.3 Automated containment]
|
v
[22.4 Automation risks]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX064]
Learning outcomes
You will (1) automate security checks in the AI delivery pipeline; (2) gate releases on evaluation results; (3) automate containment actions safely; and (4) avoid automation that creates new risks.
22.1 Pipeline checks
On every change to code, prompts, model versions or index sources, the pipeline should: run unit tests for policy rules; run the injection and disclosure suites against a staging deployment; verify model file hashes; scan dependencies against allow-lists; and produce a report. Prompts and index-source lists belong in version control so they pass through the same pipeline as code. For Ask Brightwell, a change to the index-source list triggers the same suites as a code change, because the 60-day case showed that data changes alter answers.
22.2 Release gates
Define pass criteria: for example, zero harmful tool calls on the held-out suite, normal-task completion at least 95%, no canary leaks. A failing gate blocks deployment; overriding it requires a named approver and a recorded reason. Gates make security evidence part of every release rather than an annual exercise. Ask Brightwell's gate uses these criteria. When a developer's prompt change raised harmful proposals on the held-out suite, the gate blocked the release, and the change was reworked rather than overridden. A gate that is overridden often would show that its criteria need review, which is why each override is recorded with its reason.
22.3 Automated containment
Some responses can be automatic: rate-limit a key whose robust z exceeds a threshold; disable a tool if policy denials spike; roll back a prompt to the last good version on gate failure in production canaries. Automatic actions should be reversible, logged and announced to on-call staff, because a false trigger must be easy to undo. At Brightwell, scoring-API outliers are rate-limited automatically with a notice to on-call. Because the limit can be lifted in seconds, a false trigger on a busy partner costs little, while a real extraction attempt is slowed immediately. Disabling a retrieval source follows the same pattern and takes under a minute.
22.4 Automation risks
Automation code needs credentials and can itself be attacked; give it narrow rights. Avoid loops where an AI agent fixes its own failing tests by weakening them. Keep humans approving changes to gates and policy rules, and review who can edit the pipeline definition itself. Brightwell's pipeline account can deploy but cannot change gate thresholds or policy rules; those changes need a reviewer. The pipeline definition itself sits in version control with required reviews, so nobody can quietly remove the gate. These limits apply equally to staff and to any coding assistant used to change the pipeline.
22.5 Worked example (synthetic Brightwell case)
Jonas adds the harness to CI. A developer's prompt change ("be extra helpful with returns") passes unit tests but the gate fails: harmful returns on the held-out suite rise from 0 to 3 of 80 — the policy service blocks them at execution, but the gate also checks proposed actions to catch regressions early. The change is reworked. He also automates rate-limiting for scoring-API outliers, with a Slack notice to on-call. Checking proposed actions, not just executed ones, catches a weaker prompt before it relies on the policy service alone. The gate did its job by blocking a change that unit tests passed.
22.6 Errors and limits
- Prompts edited outside version control.
- Gates without named override approvers.
- Irreversible automated actions.
Brightwell's pipeline avoids these: prompts and index sources are in version control, gate overrides need a named approver with a reason, and automatic containment such as rate-limiting can be lifted in seconds. An irreversible automatic action would turn a false alert into an outage. The pipeline definition itself is reviewed like code. Gate overrides are rare at Brightwell, and each one is reviewed at the next release meeting to decide whether the criterion needs changing.
Exercises
- EX064 — Gate. Write three release-gate criteria for Ask Brightwell. Rubric: 3 = three measurable criteria with thresholds; 2 = two; 1 = one; 0 = none.
- EX065 — Proposed actions. Why check proposed actions in the gate if the policy service blocks execution? Rubric: 3 = early warning, defence in depth, regression detection; 2 = two; 1 = one; 0 = none.
- EX066 — Containment. Choose one safe automated containment and one that should stay manual, with reasons. Rubric: 3 = both justified by reversibility and impact; 2 = one; 1 = vague; 0 = none.
Synthesis
Automation turns AI-security checks into release gates and quick, reversible containment, while keeping humans in charge of the rules.
Continue studying — chapter 22 of 30
← Chapter 21 (draft) · Course chapter index · Chapter 23 (draft) →