Chapter 28 — AI and machine-learning security: failure analysis
Lesson map
Lesson 28 published. Map, example, EX082.
flowchart TD
N0["Learning outcomes"]
N1["28.1 Kinds of causes"]
N2["28.2 Evidence and timelines"]
N3["28.3 Root-cause methods"]
N4["28.4 Blameless reviews"]
N5["28.5 Drift or attack?"]
N6["28.7b Turning causes into fixes"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX082"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["28.1 Kinds of causes"]
N2["28.2 Evidence and timelines"]
N3["28.3 Root-cause methods"]
N4["28.4 Blameless reviews"]
N5["28.5 Drift or attack?"]
N6["28.7b Turning causes into fixes"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX082"]
[Learning outcomes]
|
v
[28.1 Kinds of causes]
|
v
[28.2 Evidence and timelines]
|
v
[28.3 Root-cause methods]
|
v
[28.4 Blameless reviews]
|
v
[28.5 Drift or attack?]
|
v
[28.7b Turning causes into fixes]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX082]
Learning outcomes
You will (1) analyse AI failures and incidents to find root causes; (2) distinguish model, data, integration and process causes; (3) use timelines and evidence; and (4) write blameless post-incident reviews with actions.
28.1 Kinds of causes
AI failures can come from the model (behaviour), the data (poisoned, shifted or leaked), integration (tools, permissions, parsing) or process (review, change management, training). Most incidents have several contributing causes.
The false closure incident in the worked example below shows this: a process cause (no upload review), an integration cause (sources mixed in retrieval) and a design cause (no grounding check) all had to be present. Removing any one would have prevented or limited the harm, which is why the actions addressed all three rather than only the most visible.
28.2 Evidence and timelines
Collect logs from the application, model gateway, policy service and tools; record model, prompt and data versions; build a timeline from first cause to detection to resolution. Preserve evidence before changing systems.
In the closure incident, the first step was exporting the retrieval logs and the indexed copy of the leaflet before anyone deleted it, because the leaflet was the main evidence. The timeline then ran from upload, to overnight indexing, to the first false answer, to the receptionist's report, which showed how long the exposure lasted.
28.3 Root-cause methods
Ask "why" repeatedly until reaching causes you can act on; use fault trees for complex failures. Avoid stopping at "the model did something wrong": ask why that behaviour had impact (excessive permissions, missing confirmation).
For the closure incident, the chain went: why did patients hear the closure? The model repeated a leaflet. Why did it read the leaflet? Partner uploads were indexed without review. Why? The upload process predated the chatbot and nobody updated it when Ask Lumen began reading the same store. That last answer pointed to change management, an action LCH could take.
28.4 Blameless reviews
Focus on systems and conditions, not individuals. Record what happened, impact, causes, what went well and actions with owners and dates. Share lessons widely.
The partner who uploaded the leaflet had followed the process they were given, and the receptionist who noticed the problem was credited in the review. Treating the incident as a system failure kept staff willing to report odd chatbot answers, which was how the problem was found at all. Blame would have taught staff to stay quiet.
28.5 Drift or attack?
Changes in error rates may be drift (new data patterns) or attack. Look for clustering by source, timing and correlation with specific inputs. The monitoring lab flags a synthetic client whose query pattern differs sharply from others.
"""Lab A: spotting extraction-style abuse in a synthetic API log.
Flags clients whose query volume or input diversity is anomalous."""
import random, statistics
rng = random.Random(21)
log = []
for c in range(12):
for _ in range(rng.randint(40, 90)):
log.append((f"client{c:02d}", round(rng.gauss(0, 1), 1)))
for i in range(600): # synthetic scraper: many, grid-like inputs
log.append(("client99", round(-3 + 6*(i % 61)/60, 1)))
by = {}
for c, x in log: by.setdefault(c, []).append(x)
vol = {c: len(v) for c, v in by.items()}
med = statistics.median(vol.values()); mad = statistics.median(abs(v-med) for v in vol.values())
for c in sorted(by):
z = (vol[c]-med)/(1.4826*mad); uniq = len(set(by[c]))/len(by[c])
flag = "FLAG" if z > 3.5 else "ok"
print(f"{c} queries={vol[c]:3d} robust_z={z:6.2f} distinct_ratio={uniq:.2f} {flag}")
Expected output:
client00 queries= 50 robust_z= -0.34 distinct_ratio=0.56 ok
client01 queries= 64 robust_z= 0.45 distinct_ratio=0.52 ok
client02 queries= 81 robust_z= 1.41 distinct_ratio=0.44 ok
client03 queries= 46 robust_z= -0.56 distinct_ratio=0.52 ok
client04 queries= 52 robust_z= -0.22 distinct_ratio=0.48 ok
client05 queries= 40 robust_z= -0.90 distinct_ratio=0.57 ok
client06 queries= 42 robust_z= -0.79 distinct_ratio=0.57 ok
client07 queries= 87 robust_z= 1.74 distinct_ratio=0.48 ok
client08 queries= 56 robust_z= 0.00 distinct_ratio=0.52 ok
client09 queries= 50 robust_z= -0.34 distinct_ratio=0.62 ok
client10 queries= 68 robust_z= 0.67 distinct_ratio=0.50 ok
client11 queries= 89 robust_z= 1.85 distinct_ratio=0.42 ok
client99 queries=600 robust_z= 30.58 distinct_ratio=0.10 FLAG
In the lab, client99 sent 600 queries with a distinct ratio of 0.10, while normal clients sent 40 to 89 with ratios near 0.5; that combination of volume and repetitive pattern suggests systematic probing rather than drift. Drift would usually shift many clients a little, not one client a lot, and would not narrow the variety of queries.
28.6 Worked example (synthetic LCH case)
Several patients were told a clinic had closed. Timeline: a partner uploaded a leaflet with hidden text; it was indexed overnight; Ask Lumen repeated the closure 23 times before a receptionist reported it. Causes: partner uploads indexed without review (process), retrieval mixed partner and LCH sources (integration), no check against structured clinic data (design). Actions: review queue for partner uploads, source labelling, grounding check.
The 23 false answers were counted from conversation logs, and clinic staff phoned each affected patient to confirm their appointment.
28.7 Errors and limits
- Blaming the model or a person and stopping.
- Changing systems before preserving logs.
- Actions without owners.
Stopping at "the model hallucinated" would have left the upload process unchanged, and the next hostile leaflet would have produced the same result. Deleting the leaflet before copying it would have destroyed the evidence of how the hidden text was written. Actions without owners and dates tend to stay open indefinitely; each of the three closure actions had a named owner and a deadline.
28.7b Turning causes into fixes
Each cause found in an Ask Lumen review becomes a specific change with an owner: a missing output check becomes a validator, a confusing runbook step is rewritten, an untested rollback is rehearsed. The review is closed only when each change has been made and checked, not when the meeting ends.
For the closure incident, the review stayed open until the grounding check was tested against the original leaflet. The test showed Ask Lumen now answered with the real opening hours from the structured file and noted the conflict for review, after which the review was closed and the leaflet test added to the regression suite.
Exercises
- EX082 — Causes. Classify the leaflet incident's causes into model, data, integration and process. Rubric: 3 = correct classification; 2 = most; 1 = vague; 0 = none.
- EX083 — Lab reading. Which client is flagged and on what evidence? Rubric: 3 = client99, 600 queries, robust z 30.58, distinct 0.10; 2 = name; 1 = vague; 0 = wrong.
- EX084 — Review actions. Write two actions from the leaflet review with owners and dates. Rubric: 3 = specific, owned, dated; 2 = one; 1 = vague; 0 = none.
Synthesis
Failure analysis turns incidents into learning: evidence-based timelines, layered causes and blameless reviews with owned actions.
Continue studying — chapter 28 of 40
← Chapter 27 (draft) · Course chapter index · Chapter 29 (draft) →