Chapter 14 — AI and machine-learning security: automation
Lesson map
Lesson 14 published. Map, example, EX053.
flowchart TD
N0["Learning outcomes"]
N1["14.1 Routine checks"]
N2["14.2 Pipeline tests"]
N3["14.3 Safe automatic containment"]
N4["14.4 Securing automation"]
N5["14.4b Maintaining automation"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX053"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["14.1 Routine checks"]
N2["14.2 Pipeline tests"]
N3["14.3 Safe automatic containment"]
N4["14.4 Securing automation"]
N5["14.4b Maintaining automation"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX053"]
[Learning outcomes]
|
v
[14.1 Routine checks]
|
v
[14.2 Pipeline tests]
|
v
[14.3 Safe automatic containment]
|
v
[14.4 Securing automation]
|
v
[14.4b Maintaining automation]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX053]
Learning outcomes
You will (1) automate routine AI-security checks; (2) add security tests to deployment pipelines; (3) automate safe containment steps; (4) keep automation itself secure.
14.1 Routine checks
Automate nightly: model-file hash checks; canary searches in logs; counts of tool calls per account; expired documents in the index; limit hits; cost per day. Send a short report and alert on thresholds.
Kestrel's nightly report carries six lines. The model-file hash check covers LineWatch and any self-hosted model. A canary search runs over the previous day's HelpBot replies. Restarts per account alert above 5 per day, the threshold from Ana's script. Expired articles still in the index are listed for their owners. Accounts that hit their daily token budget are named. Total vendor cost is compared with the previous week, which would have shown the 40,000-message weekend on Saturday morning.
14.2 Pipeline tests
On each change to prompts, models, retrieval sources, tools or configuration, the pipeline runs the test suite against staging and blocks deployment if criteria fail (for example, any cross-customer action or canary leak). Overrides need a named approver and a written reason.
HelpBot's gate fails if any of the following happens on staging: a cross-customer action in any agency test, a canary phrase in any reply, an injected ticket or restart in the injection suite, or fewer than 47 of the 50 normal tasks completing. The last criterion protects customers from a fix that refuses everything. When the Wi-Fi password tool failed the gate, no override was requested; the ownership check was added and the gate passed on the next run.
14.3 Safe automatic containment
Some actions can be automatic because they are safe and reversible: rate-limit an account whose usage is a far outlier; disable a single tool when policy denials spike; drop a document from retrieval when it triggers many denials. Each automatic action notifies on-call staff and is logged.
Each Kestrel example is reversible within minutes. Rate-limiting an outlier account, like the one that sent 40,000 messages, slows it without closing it, and on-call can lift the limit. Disabling the restart tool when denials spike leaves HelpBot answering questions while ticketing still works. Dropping a document that triggers many denials removes one article from retrieval until its owner reviews it. Closing accounts or deleting data is never automatic, because neither can be undone.
14.4 Securing automation
Automation scripts need credentials; give them only the rights they need, store secrets properly, and review changes to scripts. Do not let an AI coding assistant modify security tests without human review.
"""Lab H: integrity of model artefacts with SHA-256 manifests (stdlib only).
Creates synthetic 'artefact' bytes, records a manifest, then detects a
one-byte tampering and a swapped file. No real models are involved."""
import hashlib, json
files = {"model.bin": bytes(range(256)) * 40, "tokenizer.json": b'{"vocab": ["a", "b"]}',
"config.yaml": b"layers: 4\nheads: 8\n"}
manifest = {n: hashlib.sha256(b).hexdigest() for n, b in files.items()}
print(json.dumps({k: v[:16] for k, v in manifest.items()}, indent=1))
received = dict(files)
tampered = bytearray(received["model.bin"]); tampered[5000] ^= 1
received["model.bin"] = bytes(tampered)
received["config.yaml"] = b"layers: 4\nheads: 8\ntrust_remote_code: true\n"
for n, b in sorted(received.items()):
ok = hashlib.sha256(b).hexdigest() == manifest[n]
print(f"{n:15s} {'OK' if ok else 'MISMATCH'}")
Expected output:
{
"model.bin": "e96760a87768717b",
"tokenizer.json": "f63d2ddabc61f2d9",
"config.yaml": "e5bbb570ca0fcb87"
}
config.yaml MISMATCH
model.bin MISMATCH
tokenizer.json OK
Kestrel's nightly scripts run under a dedicated account that can read logs, the index metadata and the model files, and write only to the report folder; it cannot publish articles or deploy models. Its credentials sit in the secrets store, like the router credential that Ana found in a configuration file in her first week. Changes to the HelpBot test suite need a second reviewer, because weakening a test is the easiest way to make a failing gate pass.
14.4b Maintaining automation
Review automated checks each quarter: are thresholds still right, do alerts still reach someone, and do the scripts still run against the current versions? Silent automation failures are common after upgrades.
Ana's first quarterly review found the restart alert still emailing a colleague who had moved teams, so for a month nobody had seen it. The fix was to send alerts to the on-call rota rather than to people. The same review checked that the hash script still read the new LineWatch model path after a directory change in an upgrade.
14.5 Worked example (synthetic Kestrel case)
Ana automates three nightly checks and one pipeline gate. A month later, the nightly hash check reports a mismatch on LineWatch's model file (like the lab's MISMATCH lines). The deployment log shows an approved update had not updated the stored hash. It was a process gap, not an attack — she adds the hash update to the deployment script so the two cannot diverge.
The response followed the rule for an unexplained mismatch: treat it as possible tampering until shown otherwise. LineWatch stayed on its running container while Ana compared the new file's hash with the approved release and read the deployment log. Only when both matched the approved update did she record the cause as a process gap. Tying the manifest update to deployment removed the gap without weakening the check.
14.6 Errors and limits
- Automated actions that are hard to reverse.
- Pipeline gates with easy silent overrides.
- Automation accounts with broad rights.
Automatic actions that are hard to reverse, such as closing an account, could punish a customer for a false alert. A pipeline gate with an easy silent override would have let the Wi-Fi password tool ship without its ownership check, since a deadline is always a reason to override. An automation account with broad rights becomes the most valuable credential at Kestrel, because whoever controls it can change tests, models and articles at once.
Exercises
- EX053 — Nightly checks. List five nightly checks and their thresholds. Rubric: 3 = five with thresholds; 2 = four; 1 = two; 0 = none.
- EX054 — Gate. Write gate criteria for HelpBot. Rubric: 3 = measurable criteria, override rule; 2 = criteria only; 1 = vague; 0 = none.
- EX055 — Containment. Which automatic action would you not allow, and why? Rubric: 3 = irreversible or high-impact example with reason; 2 = example only; 1 = vague; 0 = none.
- EX056 — Hash mismatch. What is the correct response to the nightly mismatch before knowing the cause? Rubric: 3 = treat as possible tampering, verify, check logs, block until resolved; 2 = partial; 1 = ignore; 0 = none.
Synthesis
Automation makes checks routine and containment fast, provided actions are reversible, gates are firm and automation's own access is limited.
Continue studying — chapter 14 of 18
← Chapter 13 (draft) · Course chapter index · Chapter 15 (draft) →