Chapter 37 — AI and machine-learning security: operations | Engineering AI and machine-learning security

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 37 published. Map, example, EX109.

flowchart TD
  N0["Learning outcomes"]
  N1["37.1 Security operations for AI"]
  N2["37.2 AI incident runbooks"]
  N3["37.3 Vulnerability management"]
  N4["37.4 Change management"]
  N5["37.5 Operational metrics"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX109"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["37.1 Security operations for AI"]
  N2["37.2 AI incident runbooks"]
  N3["37.3 Vulnerability management"]
  N4["37.4 Change management"]
  N5["37.5 Operational metrics"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX109"]

[Learning outcomes]
    |
    v
[37.1 Security operations for AI]
    |
    v
[37.2 AI incident runbooks]
    |
    v
[37.3 Vulnerability management]
    |
    v
[37.4 Change management]
    |
    v
[37.5 Operational metrics]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX109]

Learning outcomes

You will (1) run AI systems in production with security operations (monitoring, incident response, vulnerability management); (2) write runbooks for AI-specific incidents; (3) manage model and prompt changes operationally; and (4) report operational metrics.

37.1 Security operations for AI

Feed AI telemetry into the security operations centre: injection-classifier flags, policy denials, unusual tool-call rates, extraction-like query patterns, model-load verification failures and drift alarms. Define severities and owners.

Analysts in the security operations centre already watch identity, network and endpoint signals, so AI events become far more useful beside them: a burst of policy denials from one account may line up with a suspicious login. Each AI signal gets a severity, an owner and a runbook before it is switched on, because an alert nobody understands is soon ignored. Sam starts with a few high-value signals and adds more once the first ones are tuned and trusted.

37.2 AI incident runbooks

Runbooks give steps for: suspected prompt injection with action taken (contain: revoke tool credentials, disable tool; investigate: trace request IDs to source document; recover: undo actions, purge poisoned documents; learn: add to tests); suspected data poisoning (freeze training, identify affected versions by lineage, roll back to verified model); suspected extraction (throttle client, preserve logs, notify legal); model integrity failure (stop loading, verify supply chain).

Containment comes first in each runbook because it is fast and reversible. Investigation follows once harm has stopped.

37.3 Vulnerability management

Track vulnerabilities in model-serving software, libraries and connectors like any software. Track model-level weaknesses (new jailbreak techniques, newly published attacks) by monitoring sources such as MITRE ATLAS case studies, vendor advisories and research. Re-test after model or prompt updates.

Model-level weaknesses differ from software bugs: there is often no patch, only a mitigation such as a new test, a filter or a tighter tool permission. Each newly published technique is assessed against the system's own threat model and recorded as not applicable, mitigated or open. Re-testing after updates matters because a new model version may be weaker against an old attack that the previous version resisted.

37.4 Change management

Changes to models, prompts, tools and retrieval sources go through the release process with security gates, canary deployment and rollback. Emergency changes (e.g., disabling a tool) are pre-approved in runbooks.

A prompt edit or a new retrieval source can change security behaviour as much as new code, so all of them pass the same gate. Pre-approved emergency changes let the on-call engineer disable a tool at night without waiting for a meeting, while the change is still logged and reviewed the next day. The aim is speed for containment and care for everything else.

37.5 Operational metrics

Report monthly: policy denials and confirmations, injection flags, incidents by type and time to contain, evaluation-suite results per release, quota breaches, and open risks. Trends matter more than single numbers.

A rising number of confirmations may mean more legitimate work or a new injection campaign, and the trend alongside flags and incidents helps tell which. Time to contain shows whether runbooks work in practice. Evaluation results per release show whether security is improving or slipping as the system changes. Open risks keep accepted problems visible, so they are revisited rather than forgotten.

37.6 Worked example (synthetic NRS case)

At 02:10 the SOC saw a burst of Wrench proposals to create work orders on one line, all confirmed by a single night-shift account within seconds. The runbook led the on-call engineer to suspend that account's Wrench tool access, trace the proposals to a bulletin uploaded at 01:55, and find it contained instruction-like text. The account owner had approved without reading. Orders were cancelled; the bulletin source was blocked; the approval UI gained a rate limit on approvals per minute.

Every confirmation took seconds, a pattern no careful reviewer produces.

37.7 Errors and limits

AI telemetry not reaching the SOC.
No runbooks for model-specific incidents.
Emergency changes without pre-approval, causing delay.

If AI telemetry stays in an engineering dashboard, nobody watches it at 02:10. Without model-specific runbooks, responders improvise, often by switching off the whole service when disabling one tool would do. Emergency changes that need a committee lose hours while an attack continues. The incident above worked because all three were in place: the alert reached the SOC, a runbook existed and suspending access was pre-approved.

Exercises

EX109 — Runbook. Write the four phases of a runbook for suspected poisoning. Rubric: 3 = contain, investigate, recover, learn with concrete steps; 2 = three; 1 = vague; 0 = none.
EX110 — Telemetry. List five AI signals the SOC should receive. Rubric: 3 = five relevant; 2 = four; 1 = two; 0 = none.
EX111 — Incident lesson. What control would have reduced the 02:10 incident's impact earlier? Rubric: 3 = approval rate limits/anomaly alerts on approvals, source trust, highlighting; 2 = one generic; 1 = vague; 0 = none.

Synthesis

Operations keep AI security alive after release: telemetry in the SOC, AI-specific runbooks, vulnerability and change management, and trend reporting.

Continue studying — chapter 37 of 50

← Chapter 36 (draft) · Course chapter index · Chapter 38 (draft) →