Chapter 31 — AI and machine-learning security: implementation II
Lesson map
Lesson 31 published. Map, example, EX091.
flowchart TD
N0["Learning outcomes"]
N1["31.1 Monitoring and abuse detection"]
N2["31.2 Data-pipeline integrity"]
N3["31.3 Secure model loading and update"]
N4["31.4 Logging and alerting"]
N5["31.6b Testing alerts end to end"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX091"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["31.1 Monitoring and abuse detection"]
N2["31.2 Data-pipeline integrity"]
N3["31.3 Secure model loading and update"]
N4["31.4 Logging and alerting"]
N5["31.6b Testing alerts end to end"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX091"]
[Learning outcomes]
|
v
[31.1 Monitoring and abuse detection]
|
v
[31.2 Data-pipeline integrity]
|
v
[31.3 Secure model loading and update]
|
v
[31.4 Logging and alerting]
|
v
[31.6b Testing alerts end to end]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX091]
Learning outcomes
You will (1) implement monitoring and abuse detection for model services; (2) implement data-pipeline integrity controls; (3) implement secure model loading and update; and (4) integrate controls with logging and alerting.
31.1 Monitoring and abuse detection
Log per-client query volumes, input diversity, token counts, refusals and tool-call proposals. Robust statistics (median and median absolute deviation) find outliers without being skewed by the outliers themselves. The lab flags a synthetic scraper that sends a grid-like sweep of inputs typical of extraction attempts.
"""Lab A: spotting extraction-style abuse in a synthetic API log.
Flags clients whose query volume or input diversity is anomalous."""
import random, statistics
rng = random.Random(21)
log = []
for c in range(12):
for _ in range(rng.randint(40, 90)):
log.append((f"client{c:02d}", round(rng.gauss(0, 1), 1)))
for i in range(600): # synthetic scraper: many, grid-like inputs
log.append(("client99", round(-3 + 6*(i % 61)/60, 1)))
by = {}
for c, x in log: by.setdefault(c, []).append(x)
vol = {c: len(v) for c, v in by.items()}
med = statistics.median(vol.values()); mad = statistics.median(abs(v-med) for v in vol.values())
for c in sorted(by):
z = (vol[c]-med)/(1.4826*mad); uniq = len(set(by[c]))/len(by[c])
flag = "FLAG" if z > 3.5 else "ok"
print(f"{c} queries={vol[c]:3d} robust_z={z:6.2f} distinct_ratio={uniq:.2f} {flag}")
Expected output:
client00 queries= 50 robust_z= -0.34 distinct_ratio=0.56 ok
client01 queries= 64 robust_z= 0.45 distinct_ratio=0.52 ok
client02 queries= 81 robust_z= 1.41 distinct_ratio=0.44 ok
client03 queries= 46 robust_z= -0.56 distinct_ratio=0.52 ok
client04 queries= 52 robust_z= -0.22 distinct_ratio=0.48 ok
client05 queries= 40 robust_z= -0.90 distinct_ratio=0.57 ok
client06 queries= 42 robust_z= -0.79 distinct_ratio=0.57 ok
client07 queries= 87 robust_z= 1.74 distinct_ratio=0.48 ok
client08 queries= 56 robust_z= 0.00 distinct_ratio=0.52 ok
client09 queries= 50 robust_z= -0.34 distinct_ratio=0.62 ok
client10 queries= 68 robust_z= 0.67 distinct_ratio=0.50 ok
client11 queries= 89 robust_z= 1.85 distinct_ratio=0.42 ok
client99 queries=600 robust_z= 30.58 distinct_ratio=0.10 FLAG
The scraper stands out on both measures: 600 queries gives a robust z above 30, and a distinct-input ratio near one in ten shows it repeats a fixed grid of 61 values. Ordinary clients sit between 40 and 89 queries with ratios around half. Using the median and MAD keeps the scraper's own volume from raising the threshold it is judged against.
31.2 Data-pipeline integrity
Hash data at ingestion; keep immutable versions; require reviewed merges for label changes; run automated checks for duplicates, class balance shifts and suspicious feature–label correlations (possible triggers); quarantine new sources until reviewed.
Each check targets a poisoning route. Hashes and immutable versions stop silent edits. Reviewed label merges catch a contractor flipping labels. Balance and correlation checks look for the signature of a backdoor, such as a feature that predicts one class almost perfectly in new data but not in old. Quarantine gives reviewers time to sample a new source before any model learns from it, which is far cheaper than retraining afterwards.
31.3 Secure model loading and update
Load only signed artefacts whose manifest digests match; use tensor-only formats; verify on the target device; keep the previous version for rollback; record which version served each request.
The sequence is checked in order on the device: signature, then digests, then format. Loading fails closed if any step fails, and the device keeps running the previous verified version. Recording the version for each request lets investigators connect a strange output to the exact model that produced it, which matters when a supplier later reports a faulty release and NRS needs to know which inspections it affected.
31.4 Logging and alerting
Correlate request IDs across model, policy and tool services. Alerts should be specific ("client99 robust z = 30.6; distinct-input ratio 0.10") and routed to an owner with a runbook. Avoid logging sensitive prompt content in full; log hashes or redacted forms when possible, consistent with data-protection obligations.
A specific alert lets the on-call engineer judge severity at once; 'unusual activity' forces them to repeat the analysis. Runbooks state the first checks, such as confirming whether the client is a known partner, and when to escalate.
31.5 Worked example (synthetic NRS case)
NRS opened a limited TrackEye API to a partner. Sam deployed the monitoring rules: robust z-score above 3.5 on daily volume, or distinct-input ratio below 0.2 with high volume, triggers review. In the first month one partner script was flagged; it was a misconfigured retry loop, not an attack, and the partner fixed it. Sam kept the rule and documented the false positive.
A retry loop and an extraction sweep both produce high volume, so the second rule, a low distinct-input ratio, helps tell them apart: a retry loop resends the same few inputs, while extraction spreads queries systematically. Documenting the false positive gives the next reviewer a known pattern to check first.
31.6 Errors and limits
- Mean-and-standard-deviation thresholds that the outlier itself inflates.
- Logging full prompts with personal data indefinitely.
- Updates loaded without signature checks.
A mean-based threshold rises with every heavy user, so a large scraper can lift the bar above its own volume and escape the flag. Full prompts kept indefinitely become a store of personal data that is itself a target and may breach retention rules. Unsigned updates let anyone who reaches the device replace the model. Each of these appears in Sam's monitoring review as a yes-or-no question with evidence attached.
31.6b Testing alerts end to end
Each monitoring rule for Wrench and TrackEye is tested by injecting a synthetic event in staging and checking that the alert reaches the on-call engineer with enough context to act. Rules that have never fired in a test are treated as unverified.
Sam's two TrackEye API rules were tested this way before the partner went live: a synthetic client sending 600 grid-like queries in staging had to trigger the distinct-ratio rule and page the on-call engineer. The first month's real alert, the partner's misconfigured retry loop, then arrived through a path already known to work, with the client's volume and robust z-score in the message.
Exercises
- EX091 — Lab reading. What robust z-score and distinct ratio did client99 have, and why use median/MAD? Rubric: 3 = 30.58 and 0.10; robust to outlier influence; 2 = numbers only; 1 = vague; 0 = wrong.
- EX092 — Trigger check. Describe an automated check for possible backdoor triggers in new training data. Rubric: 3 = look for rare features strongly correlated with one label, compare with prior versions, flag for review; 2 = partial; 1 = vague; 0 = none.
- EX093 — Privacy in logs. How do you keep logs useful but privacy-preserving? Rubric: 3 = redact/hash content, retention limits, access control, keep metadata; 2 = two; 1 = one; 0 = none.
Synthesis
Implementation II adds the detective and integrity controls: robust monitoring, data and model integrity checks, and correlated, privacy-aware logging.
Continue studying — chapter 31 of 50
← Chapter 30 (draft) · Course chapter index · Chapter 32 (draft) →