Chapter 8 — AI and machine-learning security: failure modes

Lesson map

Lesson 08 published. Map, example, EX029.

flowchart TD
  N0["Learning outcomes"]
  N1["8.1 Failure modes"]
  N2["8.2 Accident or attack?"]
  N3["8.3 Customer effect"]
  N4["8.4 First responses"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX029"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["8.1 Failure modes"]
  N2["8.2 Accident or attack?"]
  N3["8.3 Customer effect"]
  N4["8.4 First responses"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX029"]

[Learning outcomes]
    |
    v
[8.1 Failure modes]
    |
    v
[8.2 Accident or attack?]
    |
    v
[8.3 Customer effect]
    |
    v
[8.4 First responses]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX029]

Learning outcomes

You will (1) recognise the main ways AI services fail or are attacked; (2) distinguish accidental failures from attacks; (3) describe the effect of each on customers; (4) link each failure mode to a first response.

8.1 Failure modes

• Prompt injection — instructions in messages or documents steer the assistant (LLM01).
• Disclosure — the assistant reveals data a customer should not see (LLM02).
• Excessive agency — the assistant uses tools beyond what the task needs (LLM06).
• Hallucination and misinformation — confident wrong answers (LLM09).
• Evasion — inputs crafted to fool a classifier.
• Poisoning — corrupted training data or index content.
• Extraction — repeated queries used to copy a model or recover data.
• Consumption — floods that raise costs or deny service (LLM10).
• Drift — real-world data changes so the model's accuracy degrades.

At Kestrel each mode has a case: injection, the edited restart article; disclosure, the misfiled canary; agency, cross-account restarts; drift, LineWatch.

8.2 Accident or attack?

Accidents (an outdated article, a model update, drift) usually affect many users evenly. Attacks often cluster around certain accounts, inputs or times, and show deliberate patterns such as repeated variations. Check versions and change logs first, then patterns.

Kestrel's incidents illustrate both sides. LineWatch's missed alerts affected all devices of one new router model evenly and began with a hardware rollout: drift. The factory-reset answers affected everyone who asked a related question and began with one article's publication: a content error. The 900 "not spam" reports came from one account and targeted one short link: deliberate. The three accounts with more than 50 restarts looked like abuse at first, but the change log showed the edited article, and the pattern followed whoever asked about slow speed.

8.3 Customer effect

Describe failures by what customers experience: wrong advice, lost service, exposed data, money lost. This helps prioritise and communicate.

Ana's incident notes describe effects this way. The factory-reset article meant customers lost their router configuration and had to call support. The cross-account restart flaw meant one customer could cut another's internet. The misfiled article exposed a staff-only procedure. The 40,000-message weekend doubled the vendor bill. Stated like that, the team lead could rank them without knowing what a retrieval index is, and customer support knew what to tell callers.

8.4 First responses

Injection causing actions: disable or restrict the tool. Disclosure: stop the leaking path, preserve evidence. Misinformation: correct or remove the source document. Consumption: rate-limit. Poisoning: freeze training, revert to known-good data. Extraction: limit the key.

"""Lab E: evasion of a toy linear classifier (synthetic data, stdlib only).
Trains logistic regression on two synthetic clusters, then applies a
fast-gradient-sign (FGSM-style) perturbation of size eps to every test point
and reports clean vs adversarial accuracy. Teaching model only."""
import math, random
rng = random.Random(7)
def point(label):
    cx = 1.0 if label else -1.0
    return [rng.gauss(cx, 0.7), rng.gauss(cx, 0.7)], label
train = [point(i % 2) for i in range(400)]
test = [point(i % 2) for i in range(200)]
w, b = [0.0, 0.0], 0.0
sig = lambda z: 1 / (1 + math.exp(-z))
for epoch in range(200):
    gw, gb = [0.0, 0.0], 0.0
    for x, y in train:
        p = sig(w[0]*x[0] + w[1]*x[1] + b)
        gw[0] += (p - y)*x[0]; gw[1] += (p - y)*x[1]; gb += p - y
    w = [w[i] - 0.1*gw[i]/len(train) for i in range(2)]; b -= 0.1*gb/len(train)
def acc(data, eps):
    ok = 0
    for x, y in data:
        # gradient of the loss w.r.t. x is (p - y) * w; FGSM steps along its sign
        p = sig(w[0]*x[0] + w[1]*x[1] + b)
        xa = [x[i] + eps*math.copysign(1, (p - y)*w[i]) for i in range(2)]
        ok += (sig(w[0]*xa[0] + w[1]*xa[1] + b) >= 0.5) == bool(y)
    return ok / len(data)
print(f"weights w=({w[0]:.3f}, {w[1]:.3f}) b={b:.3f}")
for eps in (0.0, 0.25, 0.5, 0.75, 1.0):
    print(f"eps={eps:.2f} accuracy={acc(test, eps):.3f}")

Expected output:

weights w=(1.795, 1.756) b=-0.080
eps=0.00 accuracy=0.985
eps=0.25 accuracy=0.950
eps=0.50 accuracy=0.840
eps=0.75 accuracy=0.660
eps=1.00 accuracy=0.505

Kestrel applied each of these. When the edited article drove restarts, the restart tool's daily cap contained harm while the article was reverted. The misfiled staff-only article was removed from the customer collection and the traces exported before anything else changed. The factory-reset article was removed and its owner told. The weekend flood was met with a per-account limit. The SpamShield feedback poisoning led to training being paused until bulk feedback was reviewed. For SpamShield's partner API, extraction is answered by limiting the offending partner's key.

8.5 Worked example (synthetic Kestrel case)

LineWatch stops alerting on a class of device faults. Ana checks: no model change; telemetry from a new router model, introduced last month, has different value ranges. That is drift, not attack. First response: route alerts for the new model to manual review; the data team retrains with new examples.

Two checks separated drift from attack here. The version history showed no change to LineWatch's model or thresholds, and the missed faults were spread evenly across every customer using the new router model rather than clustered on particular accounts. The evasion lab shows the attack version of the same symptom: accuracy fell from 0.985 to 0.840 at eps 0.5 only because each input was pushed deliberately toward the boundary. Drift moves inputs without a direction chosen by anyone.

8.6 Errors and limits

• Calling every failure an attack.
• Ignoring drift.
• Responses that fix symptoms but not causes.

Calling the LineWatch drift an attack would have sent the team hunting for an intruder while alerts stayed broken. Ignoring drift would have let the new router model go unmonitored until its first real outage. A symptom-only fix, such as lowering LineWatch's alert threshold for everyone, would have flooded operators with false alerts on the old router models without teaching the model the new value ranges. The chosen response, manual review for the new model plus retraining, addressed the cause.

Exercises

• EX029 — Classify. Classify five incidents into failure modes. Rubric: 3 = five correct; 2 = four; 1 = two or three; 0 = fewer.
• EX030 — Accident or attack. Give three signs that a HelpBot failure is an attack. Rubric: 3 = clustering, deliberate variations, suspicious source; 2 = two; 1 = one; 0 = none.
• EX031 — Evasion lab. What does accuracy 0.840 at eps 0.5 mean? Rubric: 3 = small crafted changes cause 16% errors against a 1.5% clean baseline, toy caveat; 2 = partial; 1 = vague; 0 = wrong.
• EX032 — First response. Give the first response for disclosure and for extraction. Rubric: 3 = both apt; 2 = one; 1 = vague; 0 = none.

Synthesis

Knowing failure modes, telling accidents from attacks and having a first response for each lets technicians act quickly and correctly.

Continue studying — chapter 08 of 18

← Chapter 07 (draft) · Course chapter index · Chapter 09 (draft) →