Chapter 19 — AI and machine-learning security: monitoring
Lesson map
Lesson 19 published. Map, example, EX055.
flowchart TD
N0["Learning outcomes"]
N1["19.1 Signals"]
N2["19.2 Anomaly detection"]
N3["19.3 Alert volume"]
N4["19.4 Runbooks"]
N5["19.4b Reviewing monitoring"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX055"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["19.1 Signals"]
N2["19.2 Anomaly detection"]
N3["19.3 Alert volume"]
N4["19.4 Runbooks"]
N5["19.4b Reviewing monitoring"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX055"]
[Learning outcomes]
|
v
[19.1 Signals]
|
v
[19.2 Anomaly detection]
|
v
[19.3 Alert volume]
|
v
[19.4 Runbooks]
|
v
[19.4b Reviewing monitoring]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX055]
Learning outcomes
You will (1) choose signals for monitoring AI systems; (2) build simple anomaly detection; (3) manage alert volume using base rates; and (4) write alert runbooks.
19.1 Signals
Useful signals: request and token volume per user or key; policy-service denials; tool calls per conversation; canary strings in outputs; outputs containing links; spikes in refusals or complaints; model score distributions for classifiers (drift); cost per hour. Choose signals that connect to your risk scenarios, so every alert has a reason to exist. For Ask Brightwell, Jonas links each signal to a scenario: policy denials to refund fraud, canary strings to disclosure, and token volume to cost floods. ReturnGuard's signals are scoring calls per key, which caught a compromised partner integration, and the distribution of scores over time.
19.2 Anomaly detection
Simple statistics go far. A robust z-score using median and median absolute deviation resists outliers better than mean and standard deviation. Flag users whose volume or distinct-query ratio is far from others — a pattern of scraping or model extraction.
"""Lab A: spotting extraction-style abuse in a synthetic API log.
Flags clients whose query volume or input diversity is anomalous."""
import random, statistics
rng = random.Random(21)
log = []
for c in range(12):
for _ in range(rng.randint(40, 90)):
log.append((f"client{c:02d}", round(rng.gauss(0, 1), 1)))
for i in range(600): # synthetic scraper: many, grid-like inputs
log.append(("client99", round(-3 + 6*(i % 61)/60, 1)))
by = {}
for c, x in log: by.setdefault(c, []).append(x)
vol = {c: len(v) for c, v in by.items()}
med = statistics.median(vol.values()); mad = statistics.median(abs(v-med) for v in vol.values())
for c in sorted(by):
z = (vol[c]-med)/(1.4826*mad); uniq = len(set(by[c]))/len(by[c])
flag = "FLAG" if z > 3.5 else "ok"
print(f"{c} queries={vol[c]:3d} robust_z={z:6.2f} distinct_ratio={uniq:.2f} {flag}")
Expected output:
client00 queries= 50 robust_z= -0.34 distinct_ratio=0.56 ok
client01 queries= 64 robust_z= 0.45 distinct_ratio=0.52 ok
client02 queries= 81 robust_z= 1.41 distinct_ratio=0.44 ok
client03 queries= 46 robust_z= -0.56 distinct_ratio=0.52 ok
client04 queries= 52 robust_z= -0.22 distinct_ratio=0.48 ok
client05 queries= 40 robust_z= -0.90 distinct_ratio=0.57 ok
client06 queries= 42 robust_z= -0.79 distinct_ratio=0.57 ok
client07 queries= 87 robust_z= 1.74 distinct_ratio=0.48 ok
client08 queries= 56 robust_z= 0.00 distinct_ratio=0.52 ok
client09 queries= 50 robust_z= -0.34 distinct_ratio=0.62 ok
client10 queries= 68 robust_z= 0.67 distinct_ratio=0.50 ok
client11 queries= 89 robust_z= 1.85 distinct_ratio=0.42 ok
client99 queries=600 robust_z= 30.58 distinct_ratio=0.10 FLAG
Brightwell's dashboards use this method on ReturnGuard scoring calls per account. A partner key far from the others, with queries varying one feature at a time, matched the extraction pattern from the lab. The method needs no model and runs in the same scheduled query as the dashboards.
19.3 Alert volume
Every alert costs analyst time. With rare attacks, even accurate detectors produce mostly false alarms (base-rate effect), so set thresholds by measuring alert volume on normal traffic and route low-confidence signals to dashboards rather than pagers. The transfer-volume alert in Gift Card Help, Jonas's capstone project (Chapters 29–30), showed this effect: its precision on replayed traffic was 0.5, so Jonas routed it to a dashboard. ReturnGuard's scoring-call outliers, by contrast, are handled by automatic rate-limiting with a notice to on-call rather than a page. Measuring volume on normal traffic first is what lets the team choose between these routes.
19.4 Runbooks
Each alert needs a runbook: what it means, how to check it, how to contain (rate-limit, disable a tool, roll back a prompt), whom to notify, and when to close. Runbooks let an on-call engineer act at night without guessing. The partner-key alert followed its runbook: rate-limit the key, contact the partner, review the logs. Because each step was written down, the on-call engineer did not have to decide what to do first. The same runbook names who closes the alert and what evidence is kept, so the case can be reviewed later.
19.4b Reviewing monitoring
Each month, review which alerts fired, how many were real, and which incidents were found some other way. Tune or retire noisy alerts; add signals for incidents that monitoring missed. At Brightwell, the monthly review counts each alert's real and false cases. The 60-day returns window problem was found by customers, not monitoring, and led to a regression test and a new check on retrieved document expiry. The monthly review also checks whether each alert's runbook was followed and updates it when a step was skipped or unclear.
19.5 Worked example (synthetic Brightwell case)
Jonas adds per-account dashboards. One week, a partner API key shows robust z ≈ 30 on ReturnGuard scoring calls, similar to the lab's outlier client99; the key's queries vary one feature at a time — consistent with extraction. The runbook: rate-limit the key, contact the partner, review logs. The partner's integration had been compromised; the key was rotated. The runbook turned a statistical signal into a short sequence of actions. Rotating the key ended the extraction; contacting the partner found the cause. The dashboard kept watching the replacement key, which has shown normal call volumes since.
19.6 Errors and limits
- Alerts without runbooks.
- Paging on every denial.
- Monitoring only infrastructure, not AI behaviour.
Brightwell's monitoring avoids these. Each alert has a runbook, denials go to dashboards unless they spike, and AI behaviour is watched alongside servers. The partner-key alert would have been meaningless if nobody knew what to do with it. Without behaviour signals, the injection that a policy service denies would never be seen at all. The denial dashboard covers the second, and the scoring-call alert covers the third.
Exercises
- EX055 — Signals. Choose five signals for Ask Brightwell and link each to a risk scenario. Rubric: 3 = five linked; 2 = four; 1 = two; 0 = none.
- EX056 — Robust z. Why use median and MAD rather than mean and standard deviation? Rubric: 3 = outliers inflate SD and mask themselves; 2 = partial; 1 = vague; 0 = wrong.
- EX057 — Runbook. Write the runbook for the extraction alert. Rubric: 3 = meaning, checks, containment, notification, closure; 2 = four; 1 = two; 0 = none.
Synthesis
Monitoring watches AI-specific signals, uses robust statistics, respects base rates and ties every alert to a runbook.
Continue studying — chapter 19 of 30
← Chapter 18 (draft) · Course chapter index · Chapter 20 (draft) →