Chapter 20 — AI and machine-learning security: troubleshooting
Lesson map
Lesson 20 published. Map, example, EX058.
flowchart TD
N0["Learning outcomes"]
N1["20.1 A method"]
N2["20.2 Distinguishing causes"]
N3["20.3 Safe reproduction"]
N4["20.4 Fix and prevent"]
N5["20.4b Communicating during troubleshooting"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX058"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["20.1 A method"]
N2["20.2 Distinguishing causes"]
N3["20.3 Safe reproduction"]
N4["20.4 Fix and prevent"]
N5["20.4b Communicating during troubleshooting"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX058"]
[Learning outcomes]
|
v
[20.1 A method]
|
v
[20.2 Distinguishing causes]
|
v
[20.3 Safe reproduction]
|
v
[20.4 Fix and prevent]
|
v
[20.4b Communicating during troubleshooting]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX058]
Learning outcomes
You will (1) troubleshoot AI-system failures systematically; (2) distinguish model, data, configuration and attack causes; (3) reproduce issues safely; and (4) fix and prevent recurrence.
20.1 A method
Define the symptom precisely (what, when, who, how often). Gather traces. Form hypotheses across four classes: model behaviour (a model update changed outputs), data (index contains wrong or malicious content), configuration (prompt edited, tool permission changed, limit removed) and attack (someone is deliberately causing it). Test hypotheses one at a time on a staging copy. The 60-day returns window problem followed this method: the symptom was specific, versions were checked, and retrieval traces pointed to the data class. Each class has its own first check, starting with versions.
20.2 Distinguishing causes
Check versions first: did the model, prompt, index or code change near the start of the symptom? Check retrieved content for injected text. Check whether the failures cluster around certain users or sources — attacks usually do. Check whether the issue reproduces with the previous version: if it disappears, the change is implicated. In the 60-day case, versions had not changed, which ruled out model and configuration causes quickly. Every failing answer had retrieved the same expired campaign page, so the failures clustered around a source, not around users. That clustering pattern pointed to data rather than an attack.
20.3 Safe reproduction
Reproduce on staging with synthetic data. Keep production evidence read-only. If reproduction requires dangerous inputs, keep them in an access-controlled test store. For the 60-day case, Jonas reproduced the wrong answer on staging by indexing the expired page and asking about the return window, which needed no customer data. The planted injection reviews used in later tests sit in that access-controlled store, apart from ordinary test data. Ask Brightwell's staging copies hold only synthetic customers and orders, so tests there cannot expose real personal data.
20.4 Fix and prevent
Fix the cause, add a regression test to the harness, and update runbooks. If the root cause was a design trust assumption, fix the design, not just the symptom. The 60-day case was fixed at both levels: the expired page was removed, and the design gained an expiry field and a nightly job so expired content cannot reach answers again. A regression test asks about the return window on every change. The runbook for wrong policy answers now checks retrieved documents for expired content right after the version check.
20.4b Communicating during troubleshooting
Keep a running log of what was checked and found. Tell support staff what to say to customers in the meantime, and tell owners when the cause is confirmed. During the 60-day case, support staff were told to confirm the 30-day window to any customer who asked, while Jonas checked versions and retrieval. The log recorded each check, so when the expired page was identified, the marketing owner could see exactly why it was removed and the support lead knew when to stop using the interim script.
20.5 Worked example (synthetic Brightwell case)
Customers report the assistant quoting a 60-day return window; the policy is 30 days. Jonas checks versions: no model or prompt change. He checks retrieval for the failing questions: all include a page from a marketing campaign uploaded last week ("60-day holiday returns", expired). It's a data issue, not an attack. Fix: remove the page, add an expiry field to indexed documents and a nightly job that drops expired content, plus a regression test asking about the return window. The expiry field fixes the class of problem, not just this page.
20.6 Errors and limits
- Blaming "the AI" without checking versions and data.
- Debugging in production.
- Fixes without regression tests.
The 60-day case avoided each: Jonas checked versions and data before blaming the model, reproduced on staging, and added a regression test. Had he debugged in production, a test question could have reached a real customer's conversation. Without the test, the next expired campaign page would bring the same symptom back. The runbook now lists these checks in that order: its first step is the version check, and its second is a check of retrieved documents for expired content. Checking versions and retrieval first takes minutes and rules out the commonest causes.
Exercises
- EX058 — Hypotheses. For "the assistant suddenly refuses most questions", list one hypothesis in each class. Rubric: 3 = four classes with checks; 2 = three; 1 = one or two; 0 = none.
- EX059 — Attack or not. What evidence would suggest the 60-day issue was an attack? Rubric: 3 = unknown uploader, injected text, clustering, timing; 2 = two; 1 = one; 0 = none.
- EX060 — Prevention. Propose two preventions for stale content. Rubric: 3 = expiry metadata, ownership review, regression test; 2 = two; 1 = one; 0 = none.
Synthesis
Troubleshooting AI systems separates model, data, configuration and attack causes with versions and traces, reproduces safely and prevents recurrence with tests.
Continue studying — chapter 20 of 30
← Chapter 19 (draft) · Course chapter index · Chapter 21 (draft) →