Chapter 13 — AI and machine-learning security: troubleshooting
Lesson map
Lesson 13 published. Map, example, EX049.
flowchart TD
N0["Learning outcomes"]
N1["13.1 Step by step"]
N2["13.2 Tools"]
N3["13.3 Causes"]
N4["13.4 Escalation"]
N5["13.6b Writing up the fix"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX049"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["13.1 Step by step"]
N2["13.2 Tools"]
N3["13.3 Causes"]
N4["13.4 Escalation"]
N5["13.6b Writing up the fix"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX049"]
[Learning outcomes]
|
v
[13.1 Step by step]
|
v
[13.2 Tools]
|
v
[13.3 Causes]
|
v
[13.4 Escalation]
|
v
[13.6b Writing up the fix]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX049]
Learning outcomes
You will (1) troubleshoot AI-service problems step by step; (2) use traces, versions and change logs; (3) separate data, configuration, model and attack causes; (4) escalate correctly.
13.1 Step by step
- Define the symptom: what, who, since when, how often. 2. Check recent changes: model version, prompt, retrieval sources, tools, configuration. 3. Pull traces for failing cases. 4. Form hypotheses and test them one at a time on staging. 5. Fix, verify, record.
In the factory-reset case, the steps took under a day. The symptom: customers told to factory-reset for minor issues, since two days earlier, on many accounts. Changes: no model or prompt change, but one new article. Traces: every failing answer retrieved that article. Hypothesis: the article's step 1 caused it; tested on staging by removing the article and rerunning the complaint questions, which then gave normal advice. Fix: remove, inform the owner, add a regression test, record.
13.2 Tools
Request IDs link the customer message, retrieved documents, model reply, tool calls and policy decisions. Version history shows what changed. Dashboards show when the symptom began. Comparing a failing case with a similar working one often shows the difference quickly.
Ana's comparison in the factory-reset case was a failing conversation and a working one asking about slow Wi-Fi. Same model version, same prompt version, same customer type; the only difference in the trace was the retrieved document list, where the failing one included "Advanced troubleshooting". The dashboard dated the first complaint to the article's publish time. Without request IDs joining reply, retrieval and tool calls, this comparison would have taken hours.
13.3 Causes
Data (an article is wrong, expired or malicious); configuration (a limit removed, a tool permission widened); model (vendor updated the model); attack (someone is deliberately causing it). Several causes can combine.
Kestrel's incidents cover all four. Data: the factory-reset article and the article telling HelpBot to restart routers five times. Configuration: the restart tool accepting any router, and the missing per-account gateway limit. Model: LineWatch's drift was a data change rather than a model change, but a vendor update to HelpBot's model would sit here. Attack: the 900 bulk "not spam" reports. The edited restart article combined two causes: bad content, plus a tool with no daily cap to limit its effect.
13.4 Escalation
Escalate immediately if you see signs of customer data exposure, actions on wrong accounts, or deliberate attack. Follow the incident process: preserve evidence, inform the on-call lead, do not attempt risky fixes alone.
"""Lab A: spotting extraction-style abuse in a synthetic API log.
Flags clients whose query volume or input diversity is anomalous."""
import random, statistics
rng = random.Random(21)
log = []
for c in range(12):
for _ in range(rng.randint(40, 90)):
log.append((f"client{c:02d}", round(rng.gauss(0, 1), 1)))
for i in range(600): # synthetic scraper: many, grid-like inputs
log.append(("client99", round(-3 + 6*(i % 61)/60, 1)))
by = {}
for c, x in log: by.setdefault(c, []).append(x)
vol = {c: len(v) for c, v in by.items()}
med = statistics.median(vol.values()); mad = statistics.median(abs(v-med) for v in vol.values())
for c in sorted(by):
z = (vol[c]-med)/(1.4826*mad); uniq = len(set(by[c]))/len(by[c])
flag = "FLAG" if z > 3.5 else "ok"
print(f"{c} queries={vol[c]:3d} robust_z={z:6.2f} distinct_ratio={uniq:.2f} {flag}")
Expected output:
client00 queries= 50 robust_z= -0.34 distinct_ratio=0.56 ok
client01 queries= 64 robust_z= 0.45 distinct_ratio=0.52 ok
client02 queries= 81 robust_z= 1.41 distinct_ratio=0.44 ok
client03 queries= 46 robust_z= -0.56 distinct_ratio=0.52 ok
client04 queries= 52 robust_z= -0.22 distinct_ratio=0.48 ok
client05 queries= 40 robust_z= -0.90 distinct_ratio=0.57 ok
client06 queries= 42 robust_z= -0.79 distinct_ratio=0.57 ok
client07 queries= 87 robust_z= 1.74 distinct_ratio=0.48 ok
client08 queries= 56 robust_z= 0.00 distinct_ratio=0.52 ok
client09 queries= 50 robust_z= -0.34 distinct_ratio=0.62 ok
client10 queries= 68 robust_z= 0.67 distinct_ratio=0.50 ok
client11 queries= 89 robust_z= 1.85 distinct_ratio=0.42 ok
client99 queries=600 robust_z= 30.58 distinct_ratio=0.10 FLAG
Ana's canary finding is an example of when to escalate at once: a staff-only phrase appearing in a customer reply is customer-facing data exposure, even on staging, because the same misfiling could exist in production. She exported the trace, informed the on-call lead and left the collection change to the content owner. The factory-reset case, by contrast, did not need emergency escalation once traces showed a content error, but it still went through the normal incident ticket.
13.5 Worked example (synthetic Kestrel case)
Customers report HelpBot telling them to "reset to factory settings" for minor issues — which wipes their configuration. Ana checks changes: none to model or prompt. Traces show every failing answer retrieved a new article, "Advanced troubleshooting", published two days earlier by a contractor account. The article contains the factory-reset advice in step 1. No hidden instructions — it's a content error, not an attack. She removes the article, informs the content owner, and adds a regression test. A separate dashboard check, using robust z-scores like the lab, shows no unusual account activity.
The lab's output shows what the dashboard check looks for. Twelve ordinary clients sent between 40 and 89 queries, with robust z-scores from about -0.9 to 1.9; the synthetic scraper sent 600 queries with a robust z-score of about 31 and only 10% distinct inputs, a grid-like sweep. In the factory-reset case no account came close to that, which supported the conclusion that the article was a mistake rather than a planted attack.
13.6 Errors and limits
- Changing several things at once.
- Fixing in production without staging tests.
- Not escalating possible data exposure.
Changing several things at once would have made the factory-reset fix ambiguous: had Ana removed the article and also changed the prompt, she could not have said which change fixed it. Fixing in production without staging tests risks a second outage on top of the first. Not escalating possible data exposure is the most serious of the three, because the people who must notify customers or regulators cannot act on what they are not told.
13.6b Writing up the fix
After troubleshooting, the technician records the symptom, the cause, the fix and how it was checked in the ticket. The next person who meets the same HelpBot symptom finds the answer in minutes instead of repeating the investigation.
Ana's ticket for the factory-reset case had four lines: symptom (factory-reset advice for minor issues, since publish date); cause (step 1 of the contractor's "Advanced troubleshooting" article); fix (article removed, owner informed, regression test added); check (complaint questions rerun on staging and production, normal advice returned).
The ticket also named the regression test that now asks the original complaint question.
Exercises
- EX049 — Steps. Apply the five steps to "HelpBot answers in the wrong language". Rubric: 3 = all five concretely; 2 = four; 1 = two; 0 = none.
- EX050 — Compare. How does comparing a failing and a working case help? Rubric: 3 = isolates differing retrieval, version, input; 2 = partial; 1 = vague; 0 = none.
- EX051 — Escalation. List three symptoms requiring immediate escalation. Rubric: 3 = three apt; 2 = two; 1 = one; 0 = none.
- EX052 — Monitor lab. What would a robust z of 30 for one account suggest? Rubric: 3 = far outlier, possible scraping/extraction, investigate; 2 = partial; 1 = vague; 0 = wrong.
Synthesis
Troubleshooting AI services is systematic: define, check changes, trace, test hypotheses, fix and record — escalating at once when customers may be harmed.
Continue studying — chapter 13 of 18
← Chapter 12 (draft) · Course chapter index · Chapter 14 (draft) →