◇ OpenCyber Academy
Professional · Chapter 10 · English

Chapter 10 — AI and machine-learning security: quality and troubleshooting

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 10 — AI and machine-learning security: quality and troubleshooting

Lesson map

Lesson 10 published. Map, example, EX046.

flowchart TD
  N0["Learning outcomes"]
  N1["10.1 Quality checks"]
  N2["10.2 Troubleshooting steps"]
  N3["10.3 Accident or attack"]
  N4["10.4 Preventing recurrence"]
  N5["10.5 Self-review"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX046"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["10.1 Quality checks"]
  N2["10.2 Troubleshooting steps"]
  N3["10.3 Accident or attack"]
  N4["10.4 Preventing recurrence"]
  N5["10.5 Self-review"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX046"]

[Learning outcomes]
    |
    v
[10.1 Quality checks]
    |
    v
[10.2 Troubleshooting steps]
    |
    v
[10.3 Accident or attack]
    |
    v
[10.4 Preventing recurrence]
    |
    v
[10.5 Self-review]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX046]

Learning outcomes

You will (1) check the quality of AI-security work — tests, findings, evidence; (2) troubleshoot unexpected AI behaviour systematically; (3) distinguish accidents from attacks; (4) prevent recurrence; and (5) review your own work critically.

10.1 Quality checks

Are tests repeatable? Are results measured by effects? Are suites big enough for the claims made? Are held-out cases used? Are findings reproducible by someone else? Is evidence complete and dated? A peer review against these questions catches most weaknesses.

At Meridian, Sofia asks a colleague to review her ClaimChat results against these questions before they go to the product owner. Record what the review found and what changed. Ask: does the evidence support the claim, are counts and intervals reported, are untested areas named, and could someone else repeat the test from the records alone?

10.2 Troubleshooting steps

Define the symptom precisely; check recent changes (model, prompt, documents, tools, configuration, input forms); pull traces for failing and working cases; form hypotheses; test one at a time on staging; fix; verify; record.

In the FraudFlag storm case, checking recent changes found the new form field quickly. Keep notes of each hypothesis and test, so the record shows how the cause was found. Start with what changed recently, then the inputs, then the model and tools. Test one hypothesis at a time and change only one thing at a time, so you know which change made a difference.

10.3 Accident or attack

Accidents often affect many users evenly and coincide with changes. Attacks often cluster by account, source or time and show deliberate variation. Keep both in mind until evidence separates them.

In the FraudFlag storm case, the jump affected all claims evenly after a form change, which pointed to an accident. In the garage case, relabels clustered by investigator and garage over three months, which pointed towards a deliberate pattern rather than an accident and called for a careful, confidential review. Neither conclusion was reached until the evidence supported it.

10.4 Preventing recurrence

Add a regression test; add a monitoring signal; fix the process gap that allowed it (for example, linking form changes to model reviews).

"""Lab P: label-flip poisoning and a backdoor trigger on a toy nearest-centroid
classifier (synthetic data, stdlib only). Teaching model only."""
import random
rng = random.Random(11)
def point(label, trigger=False):
    cx = 1.0 if label else -1.0
    x = [rng.gauss(cx, 0.8), rng.gauss(cx, 0.8), 0.0]
    if trigger: x[2] = 5.0          # third feature is the 'trigger'
    return x, label
clean = [point(i % 2) for i in range(400)]
test = [point(i % 2) for i in range(400)]
def fit(data):
    c = {}
    for lab in (0, 1):
        pts = [x for x, y in data if y == lab]
        c[lab] = [sum(p[i] for p in pts)/len(pts) for i in range(3)]
    return c
def pred(c, x):
    d = {k: sum((x[i]-v[i])**2 for i in range(3)) for k, v in c.items()}
    return min(d, key=d.get)
def accuracy(c, data): return sum(pred(c, x) == y for x, y in data)/len(data)
print("label-flip poisoning (flip a fraction of class-1 labels to 0):")
for frac in (0.0, 0.1, 0.2, 0.3, 0.4):
    data = [(x, y) for x, y in clean]; ones = [i for i, (x, y) in enumerate(data) if y == 1]
    for i in ones[:int(frac*len(ones))]: data[i] = (data[i][0], 0)
    print(f"  flipped={frac:.1f} clean_test_accuracy={accuracy(fit(data), test):.3f}")
print("backdoor: add n triggered class-1 points labelled 0")
trig_test = [point(1, trigger=True) for _ in range(200)]
for n in (0, 10, 40, 80):
    data = clean + [(point(1, trigger=True)[0], 0) for _ in range(n)]
    c = fit(data)
    asr = sum(pred(c, x) == 0 for x, _ in trig_test)/len(trig_test)
    print(f"  poison_points={n:3d} clean_accuracy={accuracy(c, test):.3f} attack_success={asr:.3f}")

Expected output:

label-flip poisoning (flip a fraction of class-1 labels to 0):
  flipped=0.0 clean_test_accuracy=0.965
  flipped=0.1 clean_test_accuracy=0.965
  flipped=0.2 clean_test_accuracy=0.953
  flipped=0.3 clean_test_accuracy=0.950
  flipped=0.4 clean_test_accuracy=0.945
backdoor: add n triggered class-1 points labelled 0
  poison_points=  0 clean_accuracy=0.965 attack_success=0.075
  poison_points= 10 clean_accuracy=0.963 attack_success=0.145
  poison_points= 40 clean_accuracy=0.963 attack_success=0.685
  poison_points= 80 clean_accuracy=0.968 attack_success=0.980

At Meridian, the garage case led to second review for relabels and a test set of synthetic extreme claims per garage. Record each prevention measure with an owner, and check after a quarter that it is working. Prevention should address the cause and contributing factors, not only the symptom. Add a regression test for every fix, so the same failure is caught automatically after later changes, and link it to the finding.

10.5 Self-review

Before submitting work, ask: what would a sceptical reviewer challenge? Where am I relying on assumptions? What did I not test?

Sofia's self-review of the garage case asked whether a data issue had been properly ruled out before calling it a possible insider case. Write your answers down, keep them with the work and fix what you can before the work leaves you. In the garage case, the question of whether a data issue had been ruled out is the one a sceptical reviewer would ask first, so it deserves a written answer.

10.6 Worked example (synthetic Meridian case)

FraudFlag's weekly report shows a particular garage's claims never flagged, even when amounts are extreme. Sofia considers: data issue (garage mis-coded), model issue, or attack (labels poisoned). Label history shows one investigator marking every flagged claim from that garage as "genuine" over three months — 140 relabels. The poisoning lab shows how targeted labels can create a backdoor-like blind spot while overall accuracy stays steady. It is escalated as a possible insider case; Meridian adds second review for relabels and a test set of synthetic extreme claims per garage.

10.7 Errors and limits

Each error leads to a wrong response. Assuming an attack can damage trust in colleagues, while ruling it out too early can leave an insider unchecked, as the garage case might have. A fix without a regression test can be undone by the next change. Skipping self-review lets the gaps a sceptical reviewer would find reach decision makers. Check each investigation for all three.

Exercises

Synthesis

Quality and troubleshooting go together: repeatable, effect-based work, systematic diagnosis and fixes that prevent recurrence.

Continue studying — chapter 10 of 12

← Chapter 09 (draft) · Course chapter index · Chapter 11 (draft) →