Chapter 14 — AI and machine-learning security: methods I | Technologist AI and machine-learning security

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 14 — AI and machine-learning security: methods I

Lesson map

Lesson 14 published. Map, example, EX040.

flowchart TD
  N0["Learning outcomes"]
  N1["14.1 Testing classifiers"]
  N2["14.2 Testing for prompt injection"]
  N3["14.3 Suite structure"]
  N4["14.4 Recording findings"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX040"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["14.1 Testing classifiers"]
  N2["14.2 Testing for prompt injection"]
  N3["14.3 Suite structure"]
  N4["14.4 Recording findings"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX040"]

[Learning outcomes]
    |
    v
[14.1 Testing classifiers]
    |
    v
[14.2 Testing for prompt injection]
    |
    v
[14.3 Suite structure]
    |
    v
[14.4 Recording findings]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX040]

Learning outcomes

You will (1) run basic adversarial tests on classifiers; (2) test LLM applications for direct and indirect prompt injection; (3) structure a test suite with expected results; and (4) record findings reproducibly.

14.1 Testing classifiers

For a classifier such as ReturnGuard, evasion tests ask: can small, plausible changes to a request change the score? Change features an attacker controls (return reason, timing, item value split) and measure how often a fraud case is scored as safe. Goodfellow et al. (arXiv:1412.6572) showed that even simple linear behaviour makes models sensitive to small perturbations; the evasion lab shows the same in a toy model. For ReturnGuard, Jonas's evasion tests change the features a customer controls, such as splitting one large return into several small ones or changing the stated reason. The test counts how many known fraud cases fall below the alert threshold after the change, measured against the same cases before it.

14.2 Testing for prompt injection

Direct injection: the user types instructions to override rules. Indirect injection: instructions arrive in content the model reads — reviews, emails, web pages (Greshake et al., arXiv:2302.12173). Build test documents that ask the model to call a tool with harmful parameters, reveal its system prompt, or output a link that leaks data. Vary wording, language, formatting and placement. Jonas's Ask Brightwell suite has both kinds: direct requests typed by a test customer and indirect instructions planted in reviews and policy pages on staging. The indirect cases target the returns tool, because that is where an injection turns into money. Varying wording and placement prevents a defence from passing only because it recognises one phrasing.

14.3 Suite structure

Each test case: ID, category (OWASP entry), input, setup (documents planted), expected safe behaviour, how success is judged (tool log, output pattern). Keep a development set to tune defences and a held-out set for reporting. Ask Brightwell's suite records the planted document for each indirect case and judges success from the tool log. Its held-out cases are never used while tuning the separated prompt or the policy rules, so the reported rates are not inflated by tuning on them. Setup scripts plant each case's documents on staging before the run.

14.4 Recording findings

For each failure, save the full conversation, retrieved documents, tool calls and versions. Write the finding with reproduction steps, impact and severity. When 31 indirect cases started returns, Jonas saved each full conversation, the retrieved review, the tool call and the versions in use. That record let the policy service's developer reproduce every failure on staging without asking him. The same record later showed that all 31 failures crossed the same documents-to-prompt boundary. Each finding names its owner and a retest date, so the 31 boundary failures were retested together after the fix.

14.5 Worked example (synthetic Brightwell case)

Jonas builds 120 injection cases for Ask Brightwell on staging: 40 direct, 80 indirect via planted reviews and policy pages. Graded by tool logs, 31 indirect cases start a return for an order the user did not ask about; 4 direct cases reveal the system prompt. He files two findings and keeps 40 cases aside as a held-out set. Grading by tool logs is what made the 31 indirect successes visible: in several cases the reply looked helpful while a return was started. Keeping a held-out set means later defences can be measured on cases they were not tuned on.

14.6 Errors and limits

Judging success by reading replies.
Only using public jailbreak strings.
Tuning and reporting on the same cases.

Jonas's suite avoids these. It grades from tool logs rather than replies, writes indirect cases specific to Ask Brightwell's reviews and policy pages instead of relying on public jailbreak strings, and keeps 40 cases held out from tuning. A suite built only from public strings would mostly measure whether the vendor had seen them before. Tuning on held-out cases would make reported rates look better than they are.

Exercises

EX040 — Evasion test. Design three evasion tests for ReturnGuard. Rubric: 3 = three attacker-controllable changes with measurement; 2 = two; 1 = one; 0 = none.
EX041 — Injection cases. Write three indirect-injection test documents (described, not weaponised) with expected safe behaviour. Rubric: 3 = three varied cases with expectations; 2 = two; 1 = one; 0 = none.
EX042 — Test record. List the fields of a test case record. Rubric: 3 = ID, category, input, setup, expected, grader; 2 = four; 1 = two; 0 = none.

Synthesis

Basic AI-security testing systematically varies attacker-controlled inputs, judges outcomes by effects and keeps reproducible, separated suites.

Continue studying — chapter 14 of 30

← Chapter 13 (draft) · Course chapter index · Chapter 15 (draft) →