Chapter 29 — AI and machine-learning security: operations
Lesson map
Lesson 29 published. Map, example, EX085.
flowchart TD
N0["Learning outcomes"]
N1["29.1 Monitoring"]
N2["29.2 Incident response"]
N3["29.3 Change management"]
N4["29.4 Access and accounts"]
N5["29.5 Reporting"]
N6["29.5b Exercises and drills"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX085"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["29.1 Monitoring"]
N2["29.2 Incident response"]
N3["29.3 Change management"]
N4["29.4 Access and accounts"]
N5["29.5 Reporting"]
N6["29.5b Exercises and drills"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX085"]
[Learning outcomes]
|
v
[29.1 Monitoring]
|
v
[29.2 Incident response]
|
v
[29.3 Change management]
|
v
[29.4 Access and accounts]
|
v
[29.5 Reporting]
|
v
[29.5b Exercises and drills]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX085]
Learning outcomes
You will (1) operate AI systems securely day to day; (2) monitor AI-specific signals; (3) run incident response for AI incidents; and (4) manage changes to models, prompts and data.
29.1 Monitoring
Track signals that reveal attacks or failures: injection-classifier flags, policy denials, unusual tool-call rates, per-user query volumes, cost per conversation, answer-grounding failures and drift in input or output distributions. Send them to the security operations team with clear owners.
At LCH, the policy-denial count proved the most useful early signal: a sudden rise in denied bookings usually meant someone was trying to manipulate the chatbot, even when no booking succeeded. Each signal on the dashboard has an owner in the security operations team, and alerts route to the on-call engineer outside office hours.
29.2 Incident response
Prepare runbooks. For suspected injection with impact: contain (disable the affected tool or revoke its credentials), investigate (trace the conversation, retrieved documents and tool logs), recover (undo actions, remove malicious content), learn (add tests, fix process). For suspected poisoning: stop retraining, identify affected data and models through lineage, roll back. For suspected extraction: throttle, preserve logs, involve legal.
The containment step is listed first because it can be done in minutes without understanding the cause. At LCH, disabling the booking tool still lets Ask Lumen answer questions, so patients lose only self-service booking while staff investigate, and can phone the clinic instead.
29.3 Change management
Treat models, prompts, retrieval sources and tools as configuration items. Changes pass security tests, deploy gradually and can be rolled back. Hosted models may change without notice; rerun evaluation suites on a schedule.
At LCH, the system prompt lives in the same repository as the code, so a prompt edit goes through the same review and test pipeline. The provider's model can change behind the same name, so Rina's evaluation suite runs weekly against the live model, and a drop in any metric opens a ticket even when LCH changed nothing.
29.4 Access and accounts
Review who can change prompts, upload documents, label data and administer AI services. Remove stale accounts; require multi-factor authentication for administrators.
The quarterly review at LCH found two former contractors who could still label DocSort letters and an intern account with rights to edit Ask Lumen's prompt. All three were removed. Partner upload accounts are now tied to named people rather than shared logins, so a hostile document can be traced to an individual upload and suspended alone without disrupting other partners.
29.5 Reporting
Monthly reports: incidents, denials, flags, evaluation results and open risks, with trends.
LCH's monthly report shows four numbers on the first page: incidents, policy denials, grounding failures and the latest evaluation pass rate, each with the previous three months for comparison. A rising trend in denials, even with no incidents, prompts a closer look. Open risks list an owner and target date, so the report also tracks whether fixes are being made on time, and overdue items are listed separately.
29.5b Exercises and drills
Run short drills: a simulated injected document in staging, a simulated poisoned batch, a simulated key leak. Measure time to detect and contain, and update runbooks with what you learn.
LCH's first injected-document drill took 40 minutes to contain because the runbook named the wrong tool switch; the corrected runbook was used in the real incident. Drills are cheap compared with learning the same lesson during an attack, and each one ended with a short list of runbook edits that were made the same week and checked at the next drill.
29.6 Worked example (synthetic LCH case)
On a Monday morning, monitoring showed booking cancellations via Ask Lumen at five times the usual rate. The runbook led the on-call engineer to disable the cancellation tool, trace the cancellations to conversations that had retrieved a newly uploaded partner document, and restore cancelled appointments. The document's upload account was suspended pending review. Time to contain: 18 minutes.
Restoring appointments was possible because the booking log linked each cancellation to its conversation. Without that link, staff would have had to ask every patient whether they had cancelled.
29.7 Errors and limits
- AI signals not monitored by anyone.
- No runbooks for AI-specific incidents.
- Prompt changes made directly in production.
Signals that nobody watches give no protection: the false closure incident ran for a night, through 23 false answers, because no grounding signal existed and nobody watched the answers until a receptionist reported them. Without runbooks, an on-call engineer must invent a response under pressure. Editing prompts directly in production bypasses the tests that would catch a weakened instruction or a removed safeguard, and leaves no record of what changed when something goes wrong in production.
Exercises
- EX085 — Signals. List five AI signals to monitor for Ask Lumen. Rubric: 3 = five relevant; 2 = four; 1 = two; 0 = none.
- EX086 — Runbook. Write the contain/investigate/recover/learn steps for the cancellation incident. Rubric: 3 = all four concrete; 2 = three; 1 = vague; 0 = none.
- EX087 — Hosted changes. Why rerun evaluations on a schedule for hosted models? Rubric: 3 = provider updates can change behaviour without notice; detect regressions; 2 = partial; 1 = vague; 0 = wrong.
Synthesis
Operations keep AI security working after launch: monitored AI signals, practised runbooks, controlled changes and regular reporting.
Continue studying — chapter 29 of 40
← Chapter 28 (draft) · Course chapter index · Chapter 30 (draft) →