Chapter 16 — Technical documentation and safety
Lesson map
Lesson 16 published. Map, example, EX061.
flowchart TD
N0["Learning outcomes"]
N1["16.1 Control documentation"]
N2["16.2 Runbooks"]
N3["16.3 Safety on live services"]
N4["16.4 Change records"]
N5["16.6b Reviewing documents on a schedule"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX061"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["16.1 Control documentation"]
N2["16.2 Runbooks"]
N3["16.3 Safety on live services"]
N4["16.4 Change records"]
N5["16.6b Reviewing documents on a schedule"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX061"]
[Learning outcomes]
|
v
[16.1 Control documentation]
|
v
[16.2 Runbooks]
|
v
[16.3 Safety on live services]
|
v
[16.4 Change records]
|
v
[16.6b Reviewing documents on a schedule]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX061]
Learning outcomes
You will (1) write clear technical documentation for AI-security controls; (2) write runbooks for AI incidents; (3) follow safety rules when working on live services; (4) document changes and decisions.
16.1 Control documentation
For each control: purpose (which risk it addresses), where it is configured, how it is tested, who owns it, and known limits. Example: "Restart ownership check — prevents restarting other customers' routers (LLM06). Configured in router-tool policy v2.1. Tests RST-N01–N30. Owner: service ops. Limit: does not cover phone-support restarts."
The example shows each element doing work. The purpose ties the control to the cross-account restart finding. The configuration location tells the next technician where to look. The test range RST-N01–N30 lets anyone rerun the evidence. The owner says who approves changes. The limit is the most useful line: phone-support staff restart routers through a different console, so this check does not protect that path, and nobody should assume it does.
16.2 Runbooks
Each AI incident type needs a runbook: how it is detected; triage questions; containment steps (disable tool, remove document, rate-limit, roll back prompt); evidence to preserve; whom to notify; recovery; closure. Write steps as numbered actions a tired person can follow at 3 a.m.
Kestrel's runbooks follow the incidents in this book: injected content causing actions, data disclosed in a reply, a model-file hash mismatch, and a consumption spike such as the 40,000-message weekend. Each starts with how the problem shows up, for example a nightly alert or a customer complaint, and ends with a closure step that adds a regression test, as was done after the factory-reset article.
16.3 Safety on live services
Never run attack tests against live customers. Make changes in approved windows with a rollback plan. Use two-person checks for risky changes. If unsure, stop and ask.
At Kestrel, the cross-account restart test ran only on staging with synthetic accounts, because on production it would restart real customers' routers. Changes to the restart tool's policy are made in the Tuesday maintenance window with a feature flag ready to roll back. Changing a production retrieval collection needs a second person to check the change before it is applied. Ana's stop in the canary test, when she saw the ticket tool still pointed at a real queue, is the last rule in practice.
16.4 Change records
Record each change: what, why, who approved, test results, rollback plan. When incidents happen, change records are often the fastest route to the cause.
"""Lab R: token-bucket limits against unbounded consumption (synthetic).
Simulates 60 seconds of requests from a normal client and a flooding client
against a model endpoint with a per-client token bucket."""
import random
rng = random.Random(4)
def simulate(rate_per_s, burst, reqs):
tokens, last, ok = burst, 0.0, 0
for t in reqs:
tokens = min(burst, tokens + (t - last) * rate_per_s); last = t
if tokens >= 1: tokens -= 1; ok += 1
return ok
normal = sorted(rng.uniform(0, 60) for _ in range(90))
flood = sorted(rng.uniform(0, 60) for _ in range(3000))
for rate, burst in ((5, 20), (2, 10), (1, 5)):
print(f"limit={rate}/s burst={burst:2d} normal_served={simulate(rate, burst, normal):3d}/90 "
f"flood_served={simulate(rate, burst, flood):4d}/3000")
Expected output:
limit=5/s burst=20 normal_served= 90/90 flood_served= 319/3000
limit=2/s burst=10 normal_served= 90/90 flood_served= 129/3000
limit=1/s burst= 5 normal_served= 62/90 flood_served= 64/3000
In the LineWatch hash mismatch, the deployment log answered the question in minutes: an approved change, with an approver's name, had replaced the model without updating the manifest. Without that record, Ana would have had to treat it as possible tampering for much longer. A change record for raising HelpBot's rate limit would state the old and new values, the reason, the lab result for the new values and how to revert.
16.5 Worked example (synthetic Kestrel case)
Ana writes the runbook "Injected content causing actions". Step 1: confirm with traces which document was retrieved. Step 2: disable that document's collection or the affected tool via the feature flag. Step 3: export traces and the document to the evidence store. Step 4: notify the on-call lead and content owner. Step 5: check tool logs for actions taken; reverse them. Step 6: re-enable after the document is fixed and the suite passes. In a drill, a colleague who had never seen HelpBot follows it in 12 minutes.
The drill also found one unclear step: step 2 did not say where the feature flag lived, so the runbook now names the flag and the console page.
16.6 Errors and limits
- Documentation written once and never updated.
- Runbooks only the author can follow.
- Changes without rollback plans.
Documentation written once describes HelpBot before the Wi-Fi password tool, and a technician following it would miss the newest path to a customer's router. A runbook only the author can follow fails at 3 a.m. when the author is asleep; the drill, in which a colleague new to HelpBot finished in 12 minutes, is the test against that. A change without a rollback plan turns a bad configuration into an outage while someone works out how to undo it.
16.6b Reviewing documents on a schedule
Runbooks for HelpBot and SpamShield carry a review date. A runbook past its date is flagged in the team's weekly meeting and checked against the current system; out-of-date steps on a live service are a safety risk, not just untidiness.
The review date for the injected-content runbook was moved forward when the Wi-Fi password tool was added, because step 5, checking tool logs for actions taken, had to include the new tool's log. A runbook naming only restarts and tickets would have missed it.
The review of the hash-mismatch runbook found it still named LineWatch's old model path.
Exercises
- EX061 — Control doc. Document the HelpBot token-budget control. Rubric: 3 = purpose, config, tests, owner, limits; 2 = four; 1 = two; 0 = none.
- EX062 — Runbook. Write a runbook for "customer data disclosed in a reply". Rubric: 3 = detection, triage, containment, evidence, notify, recovery, closure; 2 = five; 1 = three; 0 = none.
- EX063 — Safety. List four safety rules for working on live AI services. Rubric: 3 = four apt; 2 = three; 1 = one or two; 0 = none.
- EX064 — Change record. Write a change record for raising the rate limit. Rubric: 3 = what, why, approval, tests, rollback; 2 = four; 1 = two; 0 = none.
Synthesis
Good documentation, tested runbooks and safe working practices make AI-security controls maintainable by people other than their authors.
Continue studying — chapter 16 of 18
← Chapter 15 (draft) · Course chapter index · Chapter 17 (draft) →