Chapter 26 — AI and machine-learning security: operational handoff
Lesson map
Lesson 26 published. Map, example, EX076.
flowchart TD
N0["Learning outcomes"]
N1["26.1 What operations needs"]
N2["26.2 AI incident runbooks"]
N3["26.3 Objectives"]
N4["26.4 Training"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX076"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["26.1 What operations needs"]
N2["26.2 AI incident runbooks"]
N3["26.3 Objectives"]
N4["26.4 Training"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX076"]
[Learning outcomes]
|
v
[26.1 What operations needs]
|
v
[26.2 AI incident runbooks]
|
v
[26.3 Objectives]
|
v
[26.4 Training]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX076]
Learning outcomes
You will (1) prepare an AI system for handoff to operations; (2) write runbooks for AI-specific incidents; (3) define service-level and security-level objectives; and (4) train the receiving team.
26.1 What operations needs
Operations teams need to know: what the system does; its components and dependencies (including vendor model versions); normal behaviour (volumes, costs, denial rates); alerts and their runbooks; how to roll back a model, prompt or index; who owns what; and the known residual risks. Without this, the first incident becomes a research project. Jonas's handoff pack for Ask Brightwell covers each item, including the pinned vendor model version and the switch for disabling a retrieval source. The pack names an owner for each item, so the Ask Brightwell support team knows who to call before switching off a retrieval source.
26.2 AI incident runbooks
Write playbooks for: suspected prompt injection causing actions; data leakage through answers; model or prompt regression; cost spike; suspected model extraction; poisoned index content. Each: detection, triage questions, containment steps (disable tool, roll back prompt, remove documents, rate-limit), evidence to preserve, communication, recovery and review. For Ask Brightwell, the injection runbook was rehearsed in the tabletop: trace the proposed refund to the retrieved document, remove it, and confirm that the policy service blocked execution. The poisoned-content runbook now uses the switch for disabling a retrieval source, added after the tabletop showed that rolling back the index took too long.
26.3 Objectives
Service objectives: availability, latency, error rate. Security objectives: zero executed out-of-policy actions; canary leaks zero; time to roll back a prompt under 15 minutes; evaluation gate run on every release. Measure and review them monthly, alongside service objectives, so security is not a separate afterthought. For Ask Brightwell, the operations team reviews executed out-of-policy actions, canary leaks, prompt rollback time and gate runs each month. The tabletop added a related check: the switch for disabling a retrieval source must work in under a minute. Each objective has its own panel on the operations dashboard.
26.4 Training
Walk the operations team through a simulated incident on staging. Let them perform the rollback and containment themselves. Update runbooks from what confused them; a runbook that only its author can follow is not finished.
"""Lab R: token-bucket limits against unbounded consumption (synthetic).
Simulates 60 seconds of requests from a normal client and a flooding client
against a model endpoint with a per-client token bucket."""
import random
rng = random.Random(4)
def simulate(rate_per_s, burst, reqs):
tokens, last, ok = burst, 0.0, 0
for t in reqs:
tokens = min(burst, tokens + (t - last) * rate_per_s); last = t
if tokens >= 1: tokens -= 1; ok += 1
return ok
normal = sorted(rng.uniform(0, 60) for _ in range(90))
flood = sorted(rng.uniform(0, 60) for _ in range(3000))
for rate, burst in ((5, 20), (2, 10), (1, 5)):
print(f"limit={rate}/s burst={burst:2d} normal_served={simulate(rate, burst, normal):3d}/90 "
f"flood_served={simulate(rate, burst, flood):4d}/3000")
Expected output:
limit=5/s burst=20 normal_served= 90/90 flood_served= 319/3000
limit=2/s burst=10 normal_served= 90/90 flood_served= 129/3000
limit=1/s burst= 5 normal_served= 62/90 flood_served= 64/3000
In the Ask Brightwell tabletop, the on-call engineer performed the trace search and document removal personally. The confusing step, rolling back the index, was rewritten after the exercise. The rewritten step was tried again in the same session by a second engineer, who completed it without help, and only then was the runbook marked ready for handoff.
26.5 Worked example (synthetic Brightwell case)
Jonas hands Ask Brightwell to the operations team. In the tabletop, a simulated injected policy page causes refunds to be proposed; the on-call engineer finds the document via traces, removes it, and confirms the policy service blocked execution. They discover that rollback of the index takes 40 minutes; Jonas adds a "disable retrieval source" switch that works in under a minute. The tabletop found a problem that documents alone would not: the index rollback was too slow. The new switch gives operations a fast containment step for any poisoned source.
26.6 Errors and limits
- Handoff without rollback procedures.
- Runbooks never rehearsed.
- Security objectives no one measures.
Jonas's handoff avoided these. Rollback procedures were rehearsed in the tabletop, which is how the slow index rollback came to light, and security objectives are reviewed monthly with service objectives. Without rehearsal, the 40-minute rollback would have been discovered during a real incident. The disable switch came directly from the rehearsal, and rehearsals are now scheduled each quarter with a different on-call engineer each time, so the runbooks are tested by people who did not write them. A missed objective, such as a slow rollback, is given an owner at the next monthly review.
Exercises
- EX076 — Handoff pack. List the contents of an AI handoff pack. Rubric: 3 = seven items from the list of what operations needs; 2 = five; 1 = three; 0 = fewer.
- EX077 — Runbook. Write the containment section of the poisoned-index runbook. Rubric: 3 = disable source, remove doc, verify, preserve evidence; 2 = three; 1 = one; 0 = none.
- EX078 — Objectives. Define three security objectives with measures. Rubric: 3 = three measurable; 2 = two; 1 = one; 0 = none.
Synthesis
Handoff is complete when operations can run, contain and roll back the AI system without its builders, guided by rehearsed runbooks and measured objectives.
Continue studying — chapter 26 of 30
← Chapter 25 (draft) · Course chapter index · Chapter 27 (draft) →